Compare commits
merge into: 10235501447:master
10235501447:finalv1
10235501447:finalv2
10235501447:finalv3
10235501447:game_control
10235501447:gesture
10235501447:gesture-game
10235501447:gesture_for_chrome
10235501447:master
10235501447:screenshot
10235501447:vosk_inc
10235501447:wavesign
10235501447:web
pull from: 10235501447:web
10235501447:finalv1
10235501447:finalv2
10235501447:finalv3
10235501447:game_control
10235501447:gesture
10235501447:gesture-game
10235501447:gesture_for_chrome
10235501447:master
10235501447:screenshot
10235501447:vosk_inc
10235501447:wavesign
10235501447:web
80 changed files with 7115 additions and 809 deletions
Split View
Diff Options
-
+0 -3.gitignore
-
+0 -149README.md
-
BINimg/主平台自动化测试结果.png
-
+14 -0index.html
-
+2907 -0package-lock.json
-
+43 -0package.json
-
BINpublic/mediapipe/gesture_recognizer.task
-
+20 -0public/mediapipe/wasm/vision_wasm_internal.js
-
BINpublic/mediapipe/wasm/vision_wasm_internal.wasm
-
+20 -0public/mediapipe/wasm/vision_wasm_nosimd_internal.js
-
BINpublic/mediapipe/wasm/vision_wasm_nosimd_internal.wasm
-
BINpublic/wavecontrol1.jpg
-
BINpublic/wavecontrol2.jpg
-
BINpublic/wavecontrol3.jpg
-
BINpublic/wavecontrol4.jpg
-
BINpublic/wavecontrol5.jpg
-
BINpublic/wavecontrol6.jpg
-
BINpublic/wavecontrol7.jpg
-
BINpublic/wavecontrol8.jpg
-
BINpublic/wavecontrol9.jpg
-
+9 -0requirements.txt
-
+0 -75sprint1/sprint1-planning.md
-
+0 -10sprint1/迭代计划.md
-
BINsprint2/WaveControl-sprint2.pptx
-
+0 -120sprint2/sprint2.md
-
BINsprint3/Sprint 3 技术文档：手势识别与游戏控制.pptx
-
+0 -146sprint3/sprint3.md
-
BINsprint4/WaveControl-sprint4.pptx
-
BINsrc-py/.DS_Store
-
+84 -0src-py/VoiceController.py
-
BINsrc-py/__pycache__/VoiceController.cpython-312.pyc
-
+45 -0src-py/main.py
-
+0 -0src-py/router/__init__.py
-
BINsrc-py/router/__pycache__/__init__.cpython-312.pyc
-
BINsrc-py/router/__pycache__/ws.cpython-312.pyc
-
+260 -0src-py/router/ws.py
-
BINsrc-tauri/.DS_Store
-
+13 -0src-tauri/.gitignore
-
+32 -0src-tauri/Cargo.toml
-
+8 -0src-tauri/Info.plist
-
+3 -0src-tauri/build.rs
-
+41 -0src-tauri/capabilities/default.json
-
+17 -0src-tauri/capabilities/desktop.json
-
+58 -0src-tauri/src/lib.rs
-
+6 -0src-tauri/src/main.rs
-
+61 -0src-tauri/tauri.conf.json
-
+13 -0src-tauri/tauri.macos.conf.json
-
BINsrc/.DS_Store
-
+35 -0src/App.vue
-
+34 -0src/AppMediaPipe.vue
-
+31 -0src/components/AutoStart.vue
-
+77 -0src/components/CircleProgress.vue
-
+81 -0src/components/DevTool.vue
-
+82 -0src/components/GestureCard.vue
-
+23 -0src/components/GestureIcon.vue
-
+41 -0src/components/Menu.vue
-
+259 -0src/hand_landmark/VideoDetector.vue
-
+291 -0src/hand_landmark/detector.ts
-
+507 -0src/hand_landmark/gesture_handler.ts
-
+49 -0src/locales/en.ts
-
+18 -0src/locales/i18n.ts
-
+47 -0src/locales/zh.ts
-
+85 -0src/main.ts
-
+30 -0src/py_api.ts
-
+38 -0src/router/index.ts
-
+87 -0src/store/app.ts
-
+46 -0src/utils/subWindow.ts
-
BINsrc/view/.DS_Store
-
+348 -0src/view/mainWindow/Guide.vue
-
+367 -0src/view/mainWindow/Home.vue
-
+607 -0src/view/mainWindow/MainWindow.vue
-
+95 -0src/view/mainWindow/Update.vue
-
+93 -0src/view/subWindow/SubWindow.vue
-
+7 -0src/vite-env.d.ts
-
+30 -0tsconfig.json
-
+10 -0tsconfig.node.json
-
+43 -0vite.config.ts
-
+0 -306项目测试文档.md
-
BIN项目测试结果.xlsx
+ 0
- 3
.gitignore
View File
| @ -1,3 +0,0 @@ | |||
| node_modules/ | |||
| src-tauri/ | |||
| ~$项目测试结果.xlsx | |||
+ 0
- 149
README.md
View File
| @ -1,149 +0,0 @@ | |||
| # 📡 WaveControl 隔空手势控制系统 | |||
| > 一套融合**手势识别**与**语音控制**的非接触式人机交互系统，包含主控制平台、手语学习平台 WaveSign、赛车游戏控制模块三大子系统，致力于打造自然、高效、多场景适配的“举手即控”体验。 | |||
| ## 项目简介 | |||
| 在厨房、医疗、演讲等“无法触控”或“不便触控”的环境中，传统鼠标/键盘交互模式效率低、操作受限。**WaveControl**以此为切入点，构建了一个基于摄像头识别的隔空控制系统，融合手势识别与语音识别，实现对系统级输入（键盘/鼠标）、手语教学以及游戏控制等功能。 | |||
| 项目采用模块化架构设计，包含三大子系统： | |||
| 1. **主控制平台**：支持窗口操作、媒体控制、鼠标替代、游戏映射等功能 | |||
| 2. **赛车游戏交互系统**：实现与《Rush Rally Origins》等赛车游戏的隔空手柄交互 | |||
| 3. **WaveSign 手语通**：面向听障人群的手语学习与社区互动平台 | |||
| ## 测试信息 | |||
| 本项目配套了完整的[测试文档](./项目测试文档.md)，涵盖以下子模块： | |||
| - ✅ **主控制系统测试程序**：手势识别 → 键盘鼠标映射 → 交互反馈 | |||
| - ✅ **游戏控制测试程序**：手势实时识别 → 虚拟手柄信号发送 → 赛车游戏响应验证 | |||
| - ✅ **手语识别与打分测试程序**：摄像头实时捕捉动作 → MediaPipe评分 → UI动画与文本反馈 | |||
| - ✅ **用户交互测试**：社区发帖、点赞评论、任务管理、日程提醒等核心功能均有覆盖 | |||
| 测试方式支持浏览器调试 + 控制台运行日志追踪，若遇白屏可通过 Chrome DevTools 检查模块加载或路径引用情况。 | |||
| ## Git 分支说明 | |||
| | 分支名 | 描述 | 跳转链接 | | |||
| | -------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | | |||
| | `master` | 主分支，已完成整合，适用于演示与部署 | [🔗 master 分支](https://gitee.com/wydhhh/software-engineering/tree/master/) | | |||
| | `finalv1` | 前后端初步融合尝试，手势 → 页面响应逻辑测试阶段 | [🔗 finalv1](https://gitee.com/wydhhh/software-engineering/tree/finalv1/) | | |||
| | `finalv2` | 完成手势识别与前端事件联动，页面按钮联动测试 | [🔗 finalv2](https://gitee.com/wydhhh/software-engineering/tree/finalv2/) | | |||
| | `finalv3` | 增加语音识别、手势控制切换、音乐控制、手语平台接入、各子系统联调优化 | [🔗 finalv3](https://gitee.com/wydhhh/software-engineering/tree/finalv3/) | | |||
| | `gesture` | 手势识别逻辑独立开发模块 | [🔗 gesture](https://gitee.com/wydhhh/software-engineering/tree/gesture/) | | |||
| | `gesture_for_chrome` | 针对 Chrome 插件开发的手势控制方案（PPT翻页） | [🔗 gesture_for_chrome](https://gitee.com/wydhhh/software-engineering/tree/gesture_for_chrome/) | | |||
| | `gesture-game` | 初步游戏控制实验，控制小球移动 | [🔗 gesture-game](https://gitee.com/wydhhh/software-engineering/tree/gesture-game/) | | |||
| | `game_control` | 控制《Rush Rally Origins》赛车游戏，已支持加速转向等 | [🔗 game_control](https://gitee.com/wydhhh/software-engineering/tree/game_control/) | | |||
| | `wavesign` | 手语通子系统开发主线，包括教学评分、社区、日程等 | [🔗 wavesign](https://gitee.com/wydhhh/software-engineering/tree/wavesign/) | | |||
| | `web` | 最初的网页原型设计，UI 静态草稿 | [🔗 web](https://gitee.com/wydhhh/software-engineering/tree/web/) | | |||
| | `screenshot` | 手势控制截屏模块，用于快速抓取操作界面 | [🔗 screenshot](https://gitee.com/wydhhh/software-engineering/tree/screenshot/) | | |||
| | `vosk_inc` | 接入 VOSK 实现语音识别与实时字幕展示 | [🔗 vosk_inc](https://gitee.com/wydhhh/software-engineering/tree/vosk_inc/) | | |||
| ## 技术架构 | |||
| | 层级 | 技术方案 | | |||
| | ------------ | ------------------------------------------------------------ | | |||
| | 前端 | vue3 + TypeScript +HTML + CSS + Tailwind CSS + JavaScript + PySide2（Qt GUI） | | |||
| | 后端 | Django 4.x（主平台 + 手语通） + Python 脚本逻辑（游戏控制） | | |||
| | 手势识别 | MediaPipe Hand Landmarker | | |||
| | 虚拟设备控制 | 键盘鼠标模拟、vgamepad 虚拟手柄（XInput） | | |||
| | 数据处理 | Kalman Filter（手势抖动滤波）、SQLite3 数据库 | | |||
| ## 模块介绍 | |||
| ### 1️⃣ 主控制平台（WaveControl） | |||
| ##### ✌️ **功能特色** | |||
| - 多种预设手势操作（点击、滚动、后退等） | |||
| - 手势范围调节与自定义手势库 | |||
| - 支持语音识别辅助控制 | |||
| - 界面直观，状态反馈实时 | |||
| ##### 🎯 应用场景 | |||
| - 📺 沙发上追剧时，不再找遥控器，用手势暂停/快进 | |||
| - 👨🏫 教学/演讲中，用手势控制 PPT 流畅翻页 | |||
| - 🧑🍳 厨房做饭时，隔空查食谱不怕弄脏设备 | |||
| - 🏥 医疗/无菌操作室中，非接触式操作电脑界面 | |||
| - 🕹️ 游戏中挥手即控，沉浸感倍增 | |||
| 🖼️ **UI 示例** | |||
| - 控制主面板（准确率/响应时间/识别窗口） | |||
| - 手势管理面板（快捷操作映射设置） | |||
| ### 2️⃣ 游戏控制模块 | |||
| 🔗 项目演示：[游戏控制项目演示视频](https://www.bilibili.com/video/BV1H5gLzsEn8?vd_source=46a0e2ec60dfb8a247c96905ee47d378) | |||
| 🎮 目标：**无需实体手柄，通过摄像头即可玩赛车游戏！** | |||
| 适配游戏：Steam平台《Rush Rally Origins》及支持 Xbox手柄的其他游戏 | |||
| **实现要点：** | |||
| | 功能 | 技术说明 | | |||
| | ------------ | ----------------------------------------------------------- | | |||
| | 摄像头识别 | OpenCV + MediaPipe | | |||
| | 手势控制映射 | 👍右手拇指上扬 = 加速 👍左手 = 刹车 ✋左倾 = 左转，右倾 = 右转 | | |||
| | 虚拟手柄接口 | vgamepad + XInput | | |||
| | 抖动滤除 | Kalman 滤波器平滑动作 | | |||
| | UI反馈 | PySide2 构建调试窗口 | | |||
| ### 3️⃣ 手语通子项目：**WaveSign** | |||
| 🔗 项目演示：[手语通项目演示视频](https://www.bilibili.com/video/BV1Fig5zHEhq?vd_source=46a0e2ec60dfb8a247c96905ee47d378) | |||
| 项目定位：帮助听障人群及其家人朋友学习、练习、交流手语的综合平台 | |||
| **功能模块：** | |||
| - ✅ **手语教学与评分系统**：上传视频或用摄像头练习手语动作，系统打分反馈 | |||
| - 🗺️ **课程地图与互动练习**：任务式学习，配合卡片式巩固练习 | |||
| - 👥 **社区交流**：发帖、评论、点赞、关注等功能 | |||
| - 📅 **日程管理**：内置待办事项与日历，辅助学习安排 | |||
| - 🧭 **生活服务**： | |||
| - 出行导航（结合地图与实时提醒） | |||
| - 辅助器具推荐 | |||
| - 就业信息推送 | |||
| - 无障碍亲子/技能活动预告 | |||
| **技术实现：** | |||
| - Django + SQLite 构建用户系统与服务逻辑 | |||
| - 使用 MediaPipe 实时评分用户手语表现 | |||
| - 前端页面响应式 + 卡片式交互体验 | |||
| ## 项目成员 | |||
| 王云岱 朱子玥 杨嘉莉 | |||
| ## 📌 项目进度 | |||
| - ✅ 第一轮：系统原型 + UI设计 + 手势识别框架 | |||
| - ✅ 第二轮：主平台功能实现 + 手势控制实现 | |||
| - ✅ 第三轮：打通语音识别 + 游戏交互控制 + 手语通平台初步实现 | |||
| - ✅ 第四轮：赛车游戏交互实现 + 手语通平台完整搭建 | |||
| - 🧪 第五轮：综合测试 + 用户体验调优 + 结项演示 | |||
BIN
img/主平台自动化测试结果.png
View File
| Before | After |
|---|---|
|
|
| Width: 2036 | Height: 681 | Size: 93 KiB |
+ 14
- 0
index.html
View File
| @ -0,0 +1,14 @@ | |||
| <!doctype html> | |||
| <html lang="en" style="height: 100%;width: 100%;"> | |||
| <head> | |||
| <meta charset="UTF-8" /> | |||
| <link rel="icon" type="image/svg+xml" href="/vite.svg" /> | |||
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> | |||
| <title>Tauri + Vue + Typescript App</title> | |||
| </head> | |||
| <body style="height: 100%;width: 100%;overflow: hidden;"> | |||
| <div id="app" style="height: 100%;width: 100%"></div> | |||
| <script type="module" src="/src/main.ts"></script> | |||
| </body> | |||
| </html> | |||
+ 2907
- 0
package-lock.json
File diff suppressed because it is too large
View File
+ 43
- 0
package.json
View File
| @ -0,0 +1,43 @@ | |||
| { | |||
| "name": "tauri", | |||
| "private": true, | |||
| "version": "0.1.0", | |||
| "type": "module", | |||
| "scripts": { | |||
| "install-reqs": "npm install && pip install -r requirements.txt", | |||
| "dev": "vite", | |||
| "build:py": "pyinstaller --noconfirm --distpath src-tauri/bin/ main_win.spec", | |||
| "build:py-mac": "pyinstaller --noconfirm --distpath src-tauri/bin/ main_mac.spec", | |||
| "build:icons": "npx tauri icon public/lazyeat.png", | |||
| "build:icons-mac": "npx tauri icon public/lazyeat.png --output src-tauri/icons/mac", | |||
| "build": "vite build", | |||
| "preview": "vite preview", | |||
| "tauri": "npx tauri", | |||
| "tauri dev": "npx tauri" | |||
| }, | |||
| "dependencies": { | |||
| "@icon-park/vue-next": "^1.4.2", | |||
| "@mediapipe/tasks-vision": "^0.10.22-rc.20250304", | |||
| "@tauri-apps/api": "^2", | |||
| "@tauri-apps/plugin-autostart": "^2.2.0", | |||
| "@tauri-apps/plugin-notification": "^2.2.2", | |||
| "@tauri-apps/plugin-opener": "^2.2.6", | |||
| "@tauri-apps/plugin-store": "^2.2.0", | |||
| "@tauri-apps/plugin-window-state": "^2.2.1", | |||
| "element-plus": "^2.9.5", | |||
| "pinia": "^3.0.1", | |||
| "pinia-shared-state": "^1.0.1", | |||
| "vue": "^3.5.13", | |||
| "vue-i18n": "^9.14.4", | |||
| "vue-router": "^4.5.0" | |||
| }, | |||
| "devDependencies": { | |||
| "@tauri-apps/cli": "^2", | |||
| "@vitejs/plugin-vue": "^5.2.1", | |||
| "naive-ui": "^2.41.0", | |||
| "sass-embedded": "^1.85.1", | |||
| "typescript": "~5.6.2", | |||
| "vite": "^6.0.3", | |||
| "vue-tsc": "^2.1.10" | |||
| } | |||
| } | |||
BIN
sprint1/隔空手势识别系统Sprint1.pptx → public/mediapipe/gesture_recognizer.task
View File
+ 20
- 0
public/mediapipe/wasm/vision_wasm_internal.js
File diff suppressed because it is too large
View File
BIN
public/mediapipe/wasm/vision_wasm_internal.wasm
View File
+ 20
- 0
public/mediapipe/wasm/vision_wasm_nosimd_internal.js
File diff suppressed because it is too large
View File
BIN
public/mediapipe/wasm/vision_wasm_nosimd_internal.wasm
View File
BIN
public/wavecontrol1.jpg
View File
| Before | After |
|---|---|
|
|
| Width: 476 | Height: 476 | Size: 18 KiB |
BIN
public/wavecontrol2.jpg
View File
| Before | After |
|---|---|
|
|
| Width: 509 | Height: 509 | Size: 21 KiB |
BIN
public/wavecontrol3.jpg
View File
| Before | After |
|---|---|
|
|
| Width: 498 | Height: 498 | Size: 22 KiB |
BIN
public/wavecontrol4.jpg
View File
| Before | After |
|---|---|
|
|
| Width: 532 | Height: 532 | Size: 24 KiB |
BIN
public/wavecontrol5.jpg
View File
| Before | After |
|---|---|
|
|
| Width: 614 | Height: 614 | Size: 25 KiB |
BIN
public/wavecontrol6.jpg
View File
| Before | After |
|---|---|
|
|
| Width: 475 | Height: 475 | Size: 22 KiB |
BIN
public/wavecontrol7.jpg
View File
| Before | After |
|---|---|
|
|
| Width: 548 | Height: 548 | Size: 21 KiB |
BIN
public/wavecontrol8.jpg
View File
| Before | After |
|---|---|
|
|
| Width: 522 | Height: 522 | Size: 20 KiB |
BIN
public/wavecontrol9.jpg
View File
| Before | After |
|---|---|
|
|
| Width: 484 | Height: 484 | Size: 18 KiB |
+ 9
- 0
requirements.txt
View File
| @ -0,0 +1,9 @@ | |||
| pynput | |||
| fastapi | |||
| opencv-python | |||
| numpy | |||
| uvicorn | |||
| vosk | |||
| pygrabber | |||
| pyinstaller | |||
| sounddevice | |||
+ 0
- 75
sprint1/sprint1-planning.md
View File
| @ -1,75 +0,0 @@ | |||
| ### PPT1 标题 时间 组员 | |||
| - 标题：隔空手势识别系统Sprint1 | |||
| - 时间：[具体时间] | |||
| - 组员：[组员姓名] | |||
| ### PPT2 motivation | |||
| 在当今数字化时代，传统的鼠标键盘操作方式存在一定的局限性，如在某些场景下操作不够便捷、卫生等问题。隔空手势识别系统的出现，能够为用户提供更加自然、便捷的交互方式，填补了市场在非接触式交互领域的空白。它可以广泛应用于智能家居、智能办公、教育演示等多个场景，为用户带来全新的体验。 | |||
| ### PPT3 影响地图 | |||
| - why? | |||
| - 解决传统交互方式在特殊场景下的不便，如疫情期间减少接触、医疗场景避免交叉感染等。 | |||
| - 满足用户对更加自然、便捷交互方式的需求，提升用户体验。 | |||
| - 助力企业提升产品的科技感和竞争力，开拓新的市场领域。 | |||
| - who? | |||
| - 普通消费者：用于家庭娱乐、智能家居控制等。 | |||
| - 办公人群：在会议演示、办公操作中提高效率。 | |||
| - 教育工作者：在教学过程中进行更加生动的演示。 | |||
| - 医疗人员：在医疗操作中避免交叉感染。 | |||
| - how? | |||
| - 核心部分分工：算法团队负责手势识别和语音识别算法的开发；软件团队负责系统的整体架构和交互界面的设计；硬件团队负责传感器等硬件设备的选型和开发。 | |||
| - 采用的技术：计算机视觉技术用于手势识别，语音识别技术用于语音交互，深度学习算法提高识别的准确性和稳定性。 | |||
| - what? | |||
| - 核心功能：远程操控鼠标指针、点击操作、复杂手势识别与响应、语音识别与指令执行。 | |||
| ### PPT4 头脑风暴 | |||
| - 点子1：在游戏中使用手势进行角色控制，增强游戏的沉浸感。 | |||
| - 点子2：在商场的自助购物终端上使用手势操作，提高购物效率。 | |||
| - 点子3：在汽车驾驶中，通过手势控制车内娱乐系统和导航系统，提高驾驶安全性。 | |||
| - 点子4：在智能健身房中，使用手势控制健身设备和课程选择。 | |||
| - 点子5：在虚拟现实和增强现实场景中，使用手势进行更加自然的交互。 | |||
| ### PPT5 basic design | |||
| 系统采用分层架构设计，包括硬件层、驱动层、算法层和应用层。硬件层负责数据的采集，驱动层负责硬件设备的驱动和数据传输，算法层负责手势和语音的识别，应用层负责与用户的交互和业务逻辑的处理。 | |||
| ### PPT6 用户故事 | |||
| - card1 | |||
| - conversation1：用户在客厅看电视，想要切换频道，但不想起身找遥控器。 | |||
| - confirmation1：用户通过隔空手势轻松切换频道，无需使用遥控器。 | |||
| - card2 | |||
| - conversation2：办公人员在会议中需要展示文档，想要翻页但不想触碰鼠标。 | |||
| - confirmation2：办公人员通过手势实现文档的翻页操作，提高会议效率。 | |||
| - card3 | |||
| - conversation3：教育工作者在教学过程中，想要展示图片但不想走到电脑前操作。 | |||
| - confirmation3：教育工作者通过手势控制图片的切换和缩放，使教学更加生动。 | |||
| ### PPT7 初步开发构想 | |||
| - v1（本周四）：完成硬件设备的选型和采购，搭建开发环境，实现基本的手势识别算法。 | |||
| - v2（下周一）：完成系统的整体架构设计，实现手势识别与鼠标指针的初步关联。 | |||
| - v3（下周四）：完善手势识别的准确性和稳定性，实现语音识别功能，并进行初步的测试。 | |||
| - v4（最终）：完成系统的所有功能开发，进行全面的测试和优化，准备上线发布。 | |||
| ### PPT8 初步模块设想 | |||
| - 子模块：硬件模块、手势识别模块、语音识别模块、交互界面模块、业务逻辑模块。 | |||
| - 工作量对比：硬件模块和算法模块的工作量相对较大，交互界面模块和业务逻辑模块的工作量相对较小。可以用简单的数字比例表示，如硬件模块：30%，手势识别模块：25%，语音识别模块：20%，交互界面模块：15%，业务逻辑模块：10%。 | |||
| - 优先级：手势识别模块和硬件模块的优先级较高，需要优先开发；语音识别模块和交互界面模块的优先级次之；业务逻辑模块的优先级相对较低。 | |||
| ### PPT9 future view | |||
| 未来，我们将不断优化系统的性能和功能，拓展更多的应用场景。例如，与更多的智能家居设备进行集成，实现更加智能化的家居控制；在工业领域应用，提高生产效率和安全性等。 | |||
| ### PPT10 用户旅程 | |||
| - 优化点1：手势识别的准确性和稳定性需要进一步提高，减少误识别的情况。 | |||
| - 优化点2：语音识别的灵敏度和准确性需要优化，确保在不同环境下都能准确识别用户的语音指令。 | |||
| - 优化点3：交互界面的设计需要更加简洁直观，提高用户的操作体验。 | |||
| - 优化点4：系统的响应速度需要加快，减少用户操作的等待时间。 | |||
| - 优化点5：增加更多的手势和语音指令，满足用户多样化的需求。 | |||
| - 优化点6：提高系统的兼容性，支持更多的操作系统和硬件设备。 | |||
| ### PPT11 在迭代中开发 | |||
| - sprint planning：在每个冲刺阶段开始前，制定详细的计划，明确目标和任务。 | |||
| - daily scrum：每天进行短会，沟通工作进展和遇到的问题。 | |||
| - sprint review：在每个冲刺阶段结束后，进行评审，展示成果并收集反馈。 | |||
| - sprint retrospective：对每个冲刺阶段进行回顾和总结，分析问题并提出改进措施。 | |||
| ### PPT12 thanks | |||
| 感谢大家的聆听！ | |||
+ 0
- 10
sprint1/迭代计划.md
View File
| @ -1,10 +0,0 @@ | |||
| # 📈 迭代计划（Sprint Plan） | |||
| ## 🗓️ 项目时间线 | |||
| | 时间 | 内容说明 | | |||
| | -------- | ------------------------------------------------- | | |||
| | 本周四 | 🧩 第一次迭代演示（系统架构设计 + 原型 + UI 草稿） | | |||
| | 下周一 | 🔁 第二次迭代演示（打通手势识别 → 系统控制的闭环） | | |||
| | 下周四 | 🎙 第三次迭代演示（加入语音控制，初步功能联动） | | |||
| | 下下周三 | 🚀 第四次迭代汇报（最终系统完整交付 + 场景演示） | | |||
BIN
sprint2/WaveControl-sprint2.pptx
View File
+ 0
- 120
sprint2/sprint2.md
View File
| @ -1,120 +0,0 @@ | |||
| ## 一、本轮迭代目标 | |||
| 本 Sprint 主要目标是完成 **基于浏览器端的手势控制系统雏形**，实现如下基础功能： | |||
| - 搭建前端 Web 应用结构与 UI 页面 | |||
| - 集成 MediaPipe 手势识别模型，完成实时手势检测 | |||
| - 映射部分手势为系统控制行为（鼠标移动、点击等） | |||
| - 搭建基础手势处理逻辑与组件封装结构 | |||
| - 支持语音识别指令发送流程（准备后端对接） | |||
| ------ | |||
| ## 二、本轮主要完成内容 | |||
| | 类别 | 工作内容 | | |||
| | -------- | ------------------------------------------------------------ | | |||
| | 前端搭建 | 基于 Vite + Vue 3 + TypeScript 完成项目结构、模块划分、页面初始化 | | |||
| | 手势识别 | 集成 Google MediaPipe WASM 模型，完成实时检测与渲染 | | |||
| | 控制逻辑 | 封装 `GestureHandler`、`TriggerAction`，实现手势到系统操作的映射 | | |||
| | UI设计 | 完成仪表盘式主界面设计，支持响应式、自定义手势预览等 | | |||
| | 模型结构 | 初步梳理 `Detector` 识别流程，统一封装初始化与推理逻辑 | | |||
| | 后端准备 | 初始化 Python 接口 WebSocket 构建，支持语音识别交互 | | |||
| ------ | |||
| ## 三、系统结构设计 | |||
| ``` | |||
| [用户摄像头] | |||
| ↓ | |||
| [VideoDetector.vue] → 捕捉视频帧，传入检测器 | |||
| ↓ | |||
| [Detector.ts] → 调用 MediaPipe 识别手势 | |||
| ↓ | |||
| [GestureHandler.ts] → 映射行为（鼠标/键盘/语音） | |||
| ↓ | |||
| [TriggerAction.ts] → 向后端/系统发出控制指令 | |||
| ↓ | |||
| [Python 后端]（语音识别/扩展指令） ← WebSocket 接入 | |||
| ``` | |||
| ------ | |||
| ## 四、核心技术栈与理由 | |||
| | 技术 | 用途 | 原因 | | |||
| | ---------------- | -------------------- | ------------------------------------------ | | |||
| | Vue 3 + Vite | 前端框架 + 构建工具 | 快速开发，热更新快，Composition API 更灵活 | | |||
| | TypeScript | 增强类型约束 | 减少运行时错误，提升可维护性 | | |||
| | MediaPipe Tasks | 手势识别模型 | 体积小，支持浏览器部署，准确率高 | | |||
| | WebSocket | 前后端实时通信 | 保持实时交互流畅 | | |||
| | Python + FastAPI | 后端扩展接口（语音） | 快速搭建接口，后续支持 Py 模型运行 | | |||
| ------ | |||
| ## 五、主要功能点与说明 | |||
| ### 1. Detector 类封装 | |||
| - 初始化 WASM 模型与识别器 | |||
| - 封装 `detect()` 每帧调用逻辑 | |||
| - 支持获取 `landmarks`, `gestures`, `handedness` | |||
| ### 2. 手势识别与映射 | |||
| - 仅识别食指 → 鼠标移动 | |||
| - 食指 + 中指 → 鼠标点击 | |||
| - 食指 + 拇指捏合 → 滚动 | |||
| - 四指竖起 → 发送快捷键 | |||
| - 小指+拇指 → 启动语音识别（前端 WebSocket） | |||
| ### 3. UI 界面模块 | |||
| - 实时视频预览窗口 | |||
| - 子窗口：手势反馈 / 执行动作提示 | |||
| - 配置面板：开关检测 / 配置行为映射 | |||
| ------ | |||
| ## 六、识别代码示例（关键手势） | |||
| ```ts | |||
| if (gesture === HandGesture.ONLY_INDEX_UP) { | |||
| this.triggerAction.moveMouse(x, y) | |||
| } | |||
| if (gesture === HandGesture.INDEX_AND_THUMB_UP) { | |||
| this.triggerAction.scrollUp() | |||
| } | |||
| ``` | |||
| - 每种手势匹配后，调用封装的 WebSocket + 系统接口发送动作 | |||
| - `TriggerAction` 支持复用 | |||
| ------ | |||
| ## 七、问题与解决方案 | |||
| | 问题描述 | 解决方法 | | |||
| | --------------------------------- | ---------------------------------------------------- | | |||
| | WebSocket 连接后反复断开 | 增加连接状态判断 `readyState !== OPEN`，避免提前发送 | | |||
| | Tauri 插件调用 `invoke undefined` | 使用 `npm run tauri dev` 启动，而非普通浏览器启动 | | |||
| | 页面刷新后路由丢失（404） | Vue Router 改为 `createWebHashHistory()` 模式 | | |||
| | 模型未加载或报错提示 | 增加 loading 控制，捕捉初始化异常 | | |||
| ------ | |||
| ## 八、未来规划（Sprint 2 预告） | |||
| | 方向 | 说明 | | |||
| | ----------------- | --------------------------------------------------- | | |||
| | 多手势拓展 | 三指滚动、多指触发多功能、多手联合判断 | | |||
| | 自定义 .task 模型 | 支持用户采集数据，自定义模型替换现有 MediaPipe 模型 | | |||
| | 后端语音指令集成 | Vosk 模型识别语音，触发系统控制 | | |||
| | 设置项管理 | 实现用户自定义快捷键、手势配置界面 | | |||
BIN
sprint3/Sprint 3 技术文档：手势识别与游戏控制.pptx
View File
+ 0
- 146
sprint3/sprint3.md
View File
| @ -1,146 +0,0 @@ | |||
| # Sprint 3 技术文档 | |||
| **识别到控制闭环打通 + 支持游戏操作** | |||
| ------ | |||
| ## 一、概述 | |||
| 本阶段实现了一个运行于浏览器端的手势识别系统，结合 MediaPipe 和 WebSocket，打通从摄像头手势识别 → 系统控制（鼠标、滚动、键盘） → 控制小游戏角色的全流程。 | |||
| 新增支持多种系统控制动作和游戏动作，包括：左右移动、跳跃、组合跳跃等。 | |||
| ------ | |||
| ## 二、模块结构概览 | |||
| 模块分工如下： | |||
| - **Detector.ts** | |||
| 加载并初始化 MediaPipe 手势模型，检测手部关键点与手势状态，并调用 `GestureHandler`。 | |||
| - **GestureHandler.ts** | |||
| 将识别出的手势映射为系统操作（鼠标、滚动、键盘指令等），支持连续确认、平滑处理、动作节流。 | |||
| - **TriggerAction.ts** | |||
| 封装 WebSocket 通信，向后端发送 JSON 格式的控制命令。 | |||
| ``` | |||
| [摄像头视频流] | |||
| ↓ | |||
| [VideoDetector.vue] | |||
| - 捕捉视频帧 | |||
| - 显示实时画面 | |||
| ↓ | |||
| [Detector.ts] | |||
| - 使用 MediaPipe Tasks WASM 模型进行手势识别 | |||
| - 返回关键点、手势类型 | |||
| ↓ | |||
| [GestureHandler.ts] | |||
| - 将识别结果转为行为 | |||
| - 识别特定组合，如食指 + 拇指 → 滚动 | |||
| ↓ | |||
| [TriggerAction.ts] | |||
| - 向 WebSocket 发送控制指令 | |||
| - 控制鼠标/键盘/后端动作 | |||
| ↓ | |||
| [后端 Python] | |||
| - WebSocket 接口接收指令 | |||
| - 准备语音接口/控制中枢 | |||
| ``` | |||
| ------ | |||
| ## 三、识别手势说明（扩展后） | |||
| 每帧识别手指竖起状态（拇指到小指，0 或 1），组合为状态串，查表识别对应手势： | |||
| | 状态串 | 手势名 | 动作描述 | | |||
| | ----------- | ------------------- | ------------------------- | | |||
| | `0,1,0,0,0` | only_index_up | 鼠标移动 | | |||
| | `1,1,0,0,0` | index_and_thumb_up | 鼠标点击 | | |||
| | `0,0,1,1,1` | scroll_gesture_2 | 页面滚动 | | |||
| | `1,0,1,1,1` | scroll_gesture_2 | 页面滚动（兼容变体） | | |||
| | `0,1,1,1,1` | four_fingers_up | 发送快捷键（如播放/全屏） | | |||
| | `1,1,1,1,1` | stop_gesture | 暂停/开始识别 | | |||
| | `1,0,0,0,1` | voice_gesture_start | 启动语音识别 | | |||
| | `0,0,0,0,0` | voice_gesture_stop | 停止语音识别 | | |||
| | `0,1,1,0,0` | jump | 小人跳跃（上键） | | |||
| | `1,1,1,0,0` | rightjump | 小人右跳（右+上组合） | | |||
| | 自定义判断 | direction_right | 小人右移（长按右键） | | |||
| | 自定义判断 | delete_gesture | 小人左移（长按左键） | | |||
| ------ | |||
| ## 四、控制行为逻辑说明 | |||
| ### 鼠标控制 | |||
| - **移动**：基于食指指尖位置映射至屏幕坐标，支持平滑处理，减少抖动。 | |||
| - **点击**：食指与拇指同时上举，节流处理防止重复点击。 | |||
| - **滚动**：捏合（拇指+食指）后上下移动控制滚动方向，带阈值判断。 | |||
| ### 快捷键/系统操作 | |||
| - 四指上举 → 发送自定义按键（如全屏、暂停、下一集等）。 | |||
| - 五指上举 → 暂停/开始识别。支持进度提示。 | |||
| - 删除手势（右手拇指伸出，其余收起）→ 连续发送 Backspace/方向键。 | |||
| - 自定义 holdKey 方法支持模拟长按。 | |||
| ### 游戏控制扩展（键盘模拟） | |||
| | 手势名 | 动作 | | |||
| | --------------- | -------------------- | | |||
| | delete_gesture | 向左移动（左键长按） | | |||
| | direction_right | 向右移动（右键长按） | | |||
| | jump | 向上跳跃（上键） | | |||
| | rightjump | 右跳（右后延迟上） | | |||
| ------ | |||
| ## 五、性能与稳定性处理 | |||
| | 问题 | 解决方案 | | |||
| | ------------------ | ------------------------------------------------------ | | |||
| | 手势误判 | 连续帧确认（minGestureCount ≥ 5） | | |||
| | 鼠标抖动 | 屏幕坐标引入平滑系数 `smoothening` | | |||
| | 动作重复触发 | 节流控制点击/滚动/快捷键/方向键（如 `CLICK_INTERVAL`） | | |||
| | WebSocket 中断重连 | 异常断线后自动重连，带重试时间间隔 | | |||
| | 滚动误触 | 拇指与食指距离阈值控制是否处于“捏合”状态 | | |||
| | UI 状态提示不同步 | 使用全局 store（如 `app_store.sub_window_info`）更新 | | |||
| ------ | |||
| ## 六、技术特性总结 | |||
| - 支持 MediaPipe WASM 模型在前端本地运行，低延迟识别 | |||
| - 所有动作模块化封装，便于未来扩展新手势、新控制指令 | |||
| - 手势控制与后端通过 WebSocket 实时交互，接口稳定 | |||
| - 支持游戏场景（虚拟角色）控制，具备实际互动展示能力 | |||
| - 控制逻辑可配置：如识别区域、快捷键内容、手势灵敏度 | |||
| ------ | |||
| ## 七、后续优化建议（下一阶段） | |||
| | 方向 | 目标 | | |||
| | -------------------- | ----------------------------------------------- | | |||
| | 增加左手手势组合识别 | 允许双手协同控制，比如捏合+四指等 | | |||
| | 提升模型识别精度 | 支持定制 MediaPipe .task 文件、自采样训练 | | |||
| | UI 设置面板 | 提供手势→动作自定义、阈值调节、快捷键修改 | | |||
| | 引入语音控制 | Whisper/Vosk 实现语音命令解析，结合手势联动使用 | | |||
| | 多模式切换 | 视频控制、PPT 控制、游戏控制等可切换交互模式 | | |||
| | 场景演示/视频录制 | 准备真实交互展示视频，用于演示或上线宣传 | | |||
| ------ | |||
| ## 八、结语 | |||
| 本轮迭代成功实现了从手势识别到控制动作的完整闭环，扩展支持了游戏角色控制（多方向移动与跳跃），并在操作流畅性、准确性与系统解耦方面都取得实质进展。后续可以围绕用户可配置性、语音识别联动与多场景适配进一步扩展系统能力。 | |||
BIN
sprint4/WaveControl-sprint4.pptx
View File
BIN
src-py/.DS_Store
View File
+ 84
- 0
src-py/VoiceController.py
View File
| @ -0,0 +1,84 @@ | |||
| import json | |||
| import threading | |||
| import sounddevice as sd | |||
| import numpy as np | |||
| from vosk import Model, KaldiRecognizer | |||
| big_model_path = "big-model" | |||
| small_model_path = "model" | |||
| class VoiceController: | |||
| def __init__(self, model_type='small'): | |||
| if model_type == 'small': | |||
| self.model = Model(small_model_path) | |||
| else: | |||
| self.model = Model(big_model_path) | |||
| self.zh_text = None | |||
| self.is_recording = False | |||
| self.recognizer = KaldiRecognizer(self.model, 16000) | |||
| # 初始化输入流 | |||
| self.stream = sd.InputStream( | |||
| samplerate=16000, | |||
| channels=1, | |||
| blocksize=4096, | |||
| dtype='int16' # 对应原来 PyAudio 的 paInt16 | |||
| ) | |||
| self.stream.start() | |||
| def record_audio(self): | |||
| self.frames = [] | |||
| print("录音开始...") | |||
| # 持续录音直到标志改变 | |||
| while self.is_recording: | |||
| data, _ = self.stream.read(4096) | |||
| self.frames.append(data.tobytes()) # 转成 bytes，保持跟原先 pyaudio 一致 | |||
| def start_record_thread(self): | |||
| self.is_recording = True | |||
| threading.Thread(target=self.record_audio, daemon=True).start() | |||
| def stop_record(self): | |||
| self.is_recording = False | |||
| def transcribe_audio(self) -> str: | |||
| self.recognizer.Reset() | |||
| # 分块处理音频数据 | |||
| for chunk in self.frames: | |||
| self.recognizer.AcceptWaveform(chunk) | |||
| result = json.loads(self.recognizer.FinalResult()) | |||
| text = result.get('text', '') | |||
| text = text.replace(' ', '') | |||
| print(f"识别结果: {text}") | |||
| return text | |||
| if __name__ == '__main__': | |||
| pass | |||
| # from PyQt5.QtWidgets import QApplication, QPushButton | |||
| # | |||
| # app = QApplication([]) | |||
| # | |||
| # # 点击按钮开始录音 | |||
| # voice_controller = VoiceController() | |||
| # | |||
| # | |||
| # def btn_clicked(): | |||
| # if voice_controller.is_recording: | |||
| # voice_controller.stop_record() | |||
| # text = voice_controller.transcribe_audio() | |||
| # print(text) | |||
| # else: | |||
| # voice_controller.start_record_thread() | |||
| # | |||
| # | |||
| # btn = QPushButton('开始录音') | |||
| # btn.clicked.connect(btn_clicked) | |||
| # btn.show() | |||
| # | |||
| # app.exec_() | |||
BIN
src-py/__pycache__/VoiceController.cpython-312.pyc
View File
+ 45
- 0
src-py/main.py
View File
| @ -0,0 +1,45 @@ | |||
| import os | |||
| import sys | |||
| import uvicorn | |||
| from fastapi import FastAPI | |||
| from fastapi.middleware.cors import CORSMiddleware | |||
| from router.ws import router as ws_router | |||
| if hasattr(sys, 'frozen'): | |||
| # pyinstaller打包成exe时，sys.argv[0]的值是exe的路径 | |||
| # os.path.dirname(sys.argv[0])可以获取exe的所在目录 | |||
| # os.chdir()可以将工作目录更改为exe的所在目录 | |||
| os.chdir(os.path.dirname(sys.argv[0])) | |||
| app = FastAPI() | |||
| app.add_middleware( | |||
| CORSMiddleware, | |||
| allow_origins=["*"], | |||
| allow_credentials=True, | |||
| allow_methods=["*"], | |||
| allow_headers=["*"], | |||
| ) | |||
| # 添加 WebSocket 路由 | |||
| app.include_router(ws_router) | |||
| @app.get("/") | |||
| def read_root(): | |||
| return "ready" | |||
| @app.get("/shutdown") | |||
| def shutdown(): | |||
| import signal | |||
| import os | |||
| os.kill(os.getpid(), signal.SIGINT) | |||
| if __name__ == '__main__': | |||
| port = 62334 | |||
| print(f"Starting server at http://127.0.0.1:{port}/docs") | |||
| uvicorn.run(app, host="127.0.0.1", port=port) | |||
+ 0
- 0
src-py/router/__init__.py
View File
BIN
src-py/router/__pycache__/__init__.cpython-312.pyc
View File
BIN
src-py/router/__pycache__/ws.cpython-312.pyc
View File
+ 260
- 0
src-py/router/ws.py
View File
| @ -0,0 +1,260 @@ | |||
| import json | |||
| from dataclasses import dataclass | |||
| from typing import Dict, List, Optional, Union | |||
| from fastapi import APIRouter, WebSocket, WebSocketDisconnect | |||
| from pynput.keyboard import Controller as KeyboardController | |||
| from pynput.keyboard import Key | |||
| from pynput.mouse import Button, Controller | |||
| router = APIRouter() | |||
| # 存储活跃的WebSocket连接 | |||
| active_connection: Optional[WebSocket] = None | |||
| @dataclass | |||
| class WebSocketMessage: | |||
| """WebSocket消息数据类""" | |||
| type: str | |||
| msg: str = "" | |||
| title: str = "提示" | |||
| duration: int = 1 | |||
| data: Dict = None | |||
| def __post_init__(self): | |||
| if self.data is None: | |||
| self.data = {"x": 0, "y": 0, "key_str": ""} | |||
| def to_dict(self) -> Dict: | |||
| """将消息转换为字典格式""" | |||
| return { | |||
| "type": self.type, | |||
| "msg": self.msg, | |||
| "title": self.title, | |||
| "duration": self.duration, | |||
| "data": self.data, | |||
| } | |||
| class WebSocketMessageType: | |||
| """WebSocket消息类型常量""" | |||
| # 系统消息类型 | |||
| INFO = "info" | |||
| SUCCESS = "success" | |||
| WARNING = "warning" | |||
| ERROR = "error" | |||
| # 鼠标操作类型 | |||
| MOUSE_MOVE = "mouse_move" | |||
| MOUSE_CLICK = "mouse_click" | |||
| MOUSE_SCROLL_UP = "mouse_scroll_up" | |||
| MOUSE_SCROLL_DOWN = "mouse_scroll_down" | |||
| # 键盘操作类型 | |||
| SEND_KEYS = "send_keys" | |||
| # 语音操作类型 | |||
| VOICE_RECORD = "voice_record" | |||
| VOICE_STOP = "voice_stop" | |||
| class MessageSender: | |||
| """消息发送器,发送给前端""" | |||
| @staticmethod | |||
| async def send_message( | |||
| ws_data_type: str, msg: str, title: str = "提示", duration: int = 1 | |||
| ) -> None: | |||
| """ | |||
| 发送消息到WebSocket客户端 | |||
| Args: | |||
| ws_data_type: 消息类型 | |||
| msg: 消息内容 | |||
| title: 消息标题 | |||
| duration: 显示持续时间 | |||
| """ | |||
| if not active_connection: | |||
| return | |||
| try: | |||
| message = WebSocketMessage( | |||
| type=ws_data_type, msg=msg, title=title, duration=duration | |||
| ) | |||
| await active_connection.send_json(message.to_dict()) | |||
| except Exception as e: | |||
| print(f"发送消息失败: {e}") | |||
| class GestureHandler: | |||
| """手势控制器""" | |||
| def __init__(self): | |||
| self.keyboard = KeyboardController() | |||
| self.mouse = Controller() | |||
| def move_mouse(self, x: int, y: int) -> None: | |||
| """移动鼠标到指定位置""" | |||
| self.mouse.position = (x, y) | |||
| def click_mouse(self) -> None: | |||
| """点击鼠标左键""" | |||
| self.mouse.click(Button.left) | |||
| def scroll_up(self) -> None: | |||
| """向上滚动""" | |||
| self.mouse.scroll(0, 1) | |||
| def scroll_down(self) -> None: | |||
| """向下滚动""" | |||
| self.mouse.scroll(0, -1) | |||
| def send_keys(self, key_str: str) -> None: | |||
| """ | |||
| 发送按键事件（支持组合键） | |||
| Args: | |||
| key_str: 按键字符串（如 'ctrl+r' 或 'F11'） | |||
| """ | |||
| try: | |||
| keys = [self.__parse_key(key) for key in key_str.split("+")] | |||
| self.__execute_keys(keys) | |||
| except Exception as e: | |||
| print(f"发送按键失败: {e}") | |||
| def __parse_key(self, key_str: str) -> Union[str, Key]: | |||
| """ | |||
| 解析单个按键 | |||
| Args: | |||
| key_str: 按键字符串 | |||
| Returns: | |||
| 解析后的按键对象 | |||
| """ | |||
| key_str = key_str.strip().lower() | |||
| if hasattr(Key, key_str): | |||
| return getattr(Key, key_str) | |||
| elif len(key_str) == 1: | |||
| return key_str | |||
| elif key_str.startswith("f"): | |||
| try: | |||
| return getattr(Key, key_str) | |||
| except AttributeError: | |||
| raise ValueError(f"无效的功能键: {key_str}") | |||
| else: | |||
| raise ValueError(f"无效的按键: {key_str}") | |||
| def __execute_keys(self, keys: List[Union[str, Key]]) -> None: | |||
| """ | |||
| 执行按键序列 | |||
| Args: | |||
| keys: 按键列表 | |||
| """ | |||
| pressed_keys = [] | |||
| try: | |||
| # 按下所有键 | |||
| for key in keys: | |||
| self.keyboard.press(key) | |||
| pressed_keys.append(key) | |||
| # 释放所有键（按相反顺序） | |||
| for key in reversed(pressed_keys): | |||
| self.keyboard.release(key) | |||
| except Exception as e: | |||
| print(f"执行按键失败: {e}") | |||
| class VoiceHandler: | |||
| """语音处理控制器""" | |||
| def __init__(self): | |||
| from VoiceController import VoiceController | |||
| self.controller: Optional[VoiceController] = None | |||
| try: | |||
| self.controller = VoiceController() | |||
| except Exception as e: | |||
| print(f"语音控制器初始化失败: {e}") | |||
| async def start_recording(self, websocket: WebSocket) -> None: | |||
| """开始录音""" | |||
| if self.controller and not self.controller.is_recording: | |||
| self.controller.start_record_thread() | |||
| async def stop_recording( | |||
| self, websocket: WebSocket, gesture_handler: GestureHandler | |||
| ) -> None: | |||
| """停止录音并处理结果""" | |||
| if self.controller and self.controller.is_recording: | |||
| self.controller.stop_record() | |||
| # 获取识别结果并输入 | |||
| text = self.controller.transcribe_audio() | |||
| if text: | |||
| gesture_handler.keyboard.type(text) | |||
| gesture_handler.keyboard.tap(Key.enter) | |||
| @router.websocket("/ws_wavecontrol") | |||
| async def websocket_endpoint(websocket: WebSocket): | |||
| """WebSocket端点处理函数""" | |||
| global active_connection | |||
| await websocket.accept() | |||
| active_connection = websocket | |||
| gesture_handler = GestureHandler() | |||
| voice_handler = VoiceHandler() | |||
| try: | |||
| while True: | |||
| data_str = await websocket.receive_text() | |||
| await _handle_message(data_str, websocket, gesture_handler, voice_handler) | |||
| except WebSocketDisconnect: | |||
| active_connection = None | |||
| except Exception as e: | |||
| print(f"WebSocket处理错误: {e}") | |||
| active_connection = None | |||
| async def _handle_message( | |||
| data_str: str, | |||
| websocket: WebSocket, | |||
| gesture_handler: GestureHandler, | |||
| voice_handler: VoiceHandler, | |||
| ) -> None: | |||
| """处理WebSocket消息""" | |||
| try: | |||
| message = WebSocketMessage(**json.loads(data_str)) | |||
| data = message.data | |||
| # 处理鼠标操作 | |||
| if message.type == WebSocketMessageType.MOUSE_MOVE: | |||
| gesture_handler.move_mouse(data["x"], data["y"]) | |||
| elif message.type == WebSocketMessageType.MOUSE_CLICK: | |||
| gesture_handler.click_mouse() | |||
| elif message.type == WebSocketMessageType.MOUSE_SCROLL_UP: | |||
| gesture_handler.scroll_up() | |||
| elif message.type == WebSocketMessageType.MOUSE_SCROLL_DOWN: | |||
| gesture_handler.scroll_down() | |||
| # 处理键盘操作 | |||
| elif message.type == WebSocketMessageType.SEND_KEYS: | |||
| gesture_handler.send_keys(data["key_str"]) | |||
| # 处理语音操作 | |||
| elif message.type == WebSocketMessageType.VOICE_RECORD: | |||
| await voice_handler.start_recording(websocket) | |||
| elif message.type == WebSocketMessageType.VOICE_STOP: | |||
| await voice_handler.stop_recording(websocket, gesture_handler) | |||
| except json.JSONDecodeError: | |||
| print("无效的JSON数据") | |||
| except Exception as e: | |||
| print(f"处理消息失败: {e}") | |||
BIN
src-tauri/.DS_Store
View File
+ 13
- 0
src-tauri/.gitignore
View File
| @ -0,0 +1,13 @@ | |||
| 2 | |||
| Cargo.lock | |||
| bin/ | |||
| icons/ | |||
| # Generated by Cargo | |||
| # will have compiled files and executables | |||
| /target/ | |||
| # Generated by Tauri | |||
| # will have schema files for capabilities auto-completion | |||
| /gen/schemas | |||
+ 32
- 0
src-tauri/Cargo.toml
View File
| @ -0,0 +1,32 @@ | |||
| [package] | |||
| name = "Lazyeat" | |||
| version = "0.3.11" | |||
| description = "Lazyeat 手势识别" | |||
| authors = ["https://github.com/maplelost"] | |||
| edition = "2021" | |||
| # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html | |||
| [lib] | |||
| # The `_lib` suffix may seem redundant but it is necessary | |||
| # to make the lib name unique and wouldn't conflict with the bin name. | |||
| # This seems to be only an issue on Windows, see https://github.com/rust-lang/cargo/issues/8519 | |||
| name = "tauri_app_lib" | |||
| crate-type = ["staticlib", "cdylib", "rlib"] | |||
| [build-dependencies] | |||
| tauri-build = { version = "2", features = [] } | |||
| [dependencies] | |||
| tauri = { version = "2", features = ["devtools"] } | |||
| tauri-plugin-opener = "2" | |||
| serde = { version = "1", features = ["derive"] } | |||
| serde_json = "1" | |||
| tauri-plugin-shell = "2" | |||
| tauri-plugin-store = "2" | |||
| tauri-plugin-notification = "2" | |||
| [target.'cfg(not(any(target_os = "android", target_os = "ios")))'.dependencies] | |||
| tauri-plugin-autostart = "2" | |||
| tauri-plugin-window-state = "2" | |||
+ 8
- 0
src-tauri/Info.plist
View File
| @ -0,0 +1,8 @@ | |||
| <?xml version="1.0" encoding="UTF-8"?> | |||
| <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> | |||
| <plist version="1.0"> | |||
| <dict> | |||
| <key>NSCameraUsageDescription</key> | |||
| <string>请允许本程序访问您的摄像头</string> | |||
| </dict> | |||
| </plist> | |||
+ 3
- 0
src-tauri/build.rs
View File
| @ -0,0 +1,3 @@ | |||
| fn main() { | |||
| tauri_build::build() | |||
| } | |||
+ 41
- 0
src-tauri/capabilities/default.json
View File
| @ -0,0 +1,41 @@ | |||
| { | |||
| "$schema": "../gen/schemas/desktop-schema.json", | |||
| "identifier": "default", | |||
| "description": "Capability for the main window", | |||
| "windows": ["main", "NewWindow_2"], | |||
| "permissions": [ | |||
| "core:default", | |||
| "shell:default", | |||
| "shell:allow-execute", | |||
| "shell:allow-spawn", | |||
| "core:path:default", | |||
| "core:event:default", | |||
| "core:window:default", | |||
| "core:app:default", | |||
| "core:resources:default", | |||
| "core:menu:default", | |||
| "core:tray:default", | |||
| "core:window:allow-destroy", | |||
| "core:window:allow-set-title", | |||
| "store:default", | |||
| "notification:default", | |||
| "notification:allow-is-permission-granted", | |||
| "notification:allow-notify", | |||
| "notification:allow-show", | |||
| "notification:allow-request-permission", | |||
| "opener:default", | |||
| "opener:allow-open-path", | |||
| "opener:allow-reveal-item-in-dir", | |||
| "opener:allow-default-urls", | |||
| "opener:allow-open-url", | |||
| "core:webview:default", | |||
| "core:window:allow-show", | |||
| "core:window:allow-hide", | |||
| "core:webview:allow-create-webview-window", | |||
| "core:window:allow-set-position", | |||
| "core:window:allow-set-size" | |||
| ] | |||
| } | |||
+ 17
- 0
src-tauri/capabilities/desktop.json
View File
| @ -0,0 +1,17 @@ | |||
| { | |||
| "identifier": "desktop-capability", | |||
| "platforms": [ | |||
| "macOS", | |||
| "windows", | |||
| "linux" | |||
| ], | |||
| "windows": [ | |||
| "main" | |||
| ], | |||
| "permissions": [ | |||
| "autostart:allow-enable", | |||
| "autostart:allow-disable", | |||
| "autostart:allow-is-enabled", | |||
| "window-state:default" | |||
| ] | |||
| } | |||
+ 58
- 0
src-tauri/src/lib.rs
View File
| @ -0,0 +1,58 @@ | |||
| // Learn more about Tauri commands at https://tauri.app/develop/calling-rust/ | |||
| #[tauri::command] | |||
| fn greet(name: &str) -> String { | |||
| format!("Hello, {}! You've been greeted from Rust!", name) | |||
| } | |||
| // 提取sidecar启动逻辑到单独的函数 | |||
| async fn start_sidecar(app: tauri::AppHandle) -> Result<String, String> { | |||
| let sidecar = app | |||
| .shell() | |||
| .sidecar("Lazyeat Backend") | |||
| .map_err(|e| format!("无法找到sidecar: {}", e))?; | |||
| let (_rx, _child) = sidecar | |||
| .spawn() | |||
| .map_err(|e| format!("无法启动sidecar: {}", e))?; | |||
| Ok("Sidecar已启动".to_string()) | |||
| } | |||
| // 保留命令供可能的手动调用 | |||
| #[tauri::command] | |||
| async fn run_sidecar(app: tauri::AppHandle) -> Result<String, String> { | |||
| start_sidecar(app).await | |||
| } | |||
| use tauri_plugin_autostart::MacosLauncher; | |||
| use tauri_plugin_autostart::ManagerExt; | |||
| use tauri_plugin_shell::process::CommandEvent; | |||
| use tauri_plugin_shell::ShellExt; | |||
| #[cfg_attr(mobile, tauri::mobile_entry_point)] | |||
| pub fn run() { | |||
| tauri::Builder::default() | |||
| .plugin(tauri_plugin_notification::init()) | |||
| // .plugin(tauri_plugin_window_state::Builder::new().build()) // 窗口状态管理,启用了导致 sub-window 无法设置decorations | |||
| .plugin(tauri_plugin_store::Builder::new().build()) | |||
| .plugin(tauri_plugin_autostart::init( | |||
| MacosLauncher::LaunchAgent, | |||
| Some(vec!["--flag1", "--flag2"]), | |||
| )) | |||
| .plugin(tauri_plugin_shell::init()) | |||
| .plugin(tauri_plugin_opener::init()) | |||
| .setup(|app| { | |||
| // 在应用启动时自动启动sidecar | |||
| let app_handle = app.handle().clone(); | |||
| tauri::async_runtime::spawn(async move { | |||
| match start_sidecar(app_handle).await { | |||
| Ok(msg) => println!("{}", msg), | |||
| Err(e) => eprintln!("启动sidecar失败: {}", e), | |||
| } | |||
| }); | |||
| Ok(()) | |||
| }) | |||
| .invoke_handler(tauri::generate_handler![greet, run_sidecar]) | |||
| .run(tauri::generate_context!()) | |||
| .expect("error while running tauri application"); | |||
| } | |||
+ 6
- 0
src-tauri/src/main.rs
View File
| @ -0,0 +1,6 @@ | |||
| // Prevents additional console window on Windows in release, DO NOT REMOVE!! | |||
| #![cfg_attr(not(debug_assertions), windows_subsystem = "windows")] | |||
| fn main() { | |||
| tauri_app_lib::run() | |||
| } | |||
+ 61
- 0
src-tauri/tauri.conf.json
View File
| @ -0,0 +1,61 @@ | |||
| { | |||
| "$schema": "https://schema.tauri.app/config/2", | |||
| "productName": "Lazyeat", | |||
| "version": "0.3.11", | |||
| "identifier": "com.Lazyeat.maplelost", | |||
| "build": { | |||
| "beforeDevCommand": "npm run dev", | |||
| "devUrl": "http://localhost:1420", | |||
| "beforeBuildCommand": "npm run build", | |||
| "frontendDist": "../dist" | |||
| }, | |||
| "app": { | |||
| "windows": [ | |||
| { | |||
| "title": "Lazyeat", | |||
| "width": 800, | |||
| "height": 600, | |||
| "devtools": true | |||
| } | |||
| ], | |||
| "security": { | |||
| "csp": null | |||
| } | |||
| }, | |||
| "bundle": { | |||
| "active": true, | |||
| "externalBin": ["bin/backend-py/Lazyeat Backend"], | |||
| "resources": { | |||
| "bin/backend-py/_internal": "_internal/", | |||
| "../model": "model/", | |||
| "../debug backend.bat": "/" | |||
| }, | |||
| "macOS": { | |||
| "dmg": { | |||
| "appPosition": { | |||
| "x": 180, | |||
| "y": 170 | |||
| }, | |||
| "applicationFolderPosition": { | |||
| "x": 480, | |||
| "y": 170 | |||
| }, | |||
| "windowSize": { | |||
| "height": 400, | |||
| "width": 660 | |||
| } | |||
| }, | |||
| "files": {}, | |||
| "hardenedRuntime": true, | |||
| "minimumSystemVersion": "10.13" | |||
| }, | |||
| "targets": ["dmg", "msi"], | |||
| "icon": [ | |||
| "icons/32x32.png", | |||
| "icons/128x128.png", | |||
| "icons/128x128@2x.png", | |||
| "icons/mac/icon.icns", | |||
| "icons/icon.ico" | |||
| ] | |||
| } | |||
| } | |||
+ 13
- 0
src-tauri/tauri.macos.conf.json
View File
| @ -0,0 +1,13 @@ | |||
| { | |||
| "$schema": "https://schema.tauri.app/config/2", | |||
| "identifier": "com.Lazyeat.maplelost", | |||
| "bundle": { | |||
| "resources": [], | |||
| "macOS": { | |||
| "files": { | |||
| "Resources/model": "../model", | |||
| "Frameworks": "./bin/backend-py/_internal" | |||
| } | |||
| } | |||
| } | |||
| } | |||
BIN
src/.DS_Store
View File
+ 35
- 0
src/App.vue
View File
| @ -0,0 +1,35 @@ | |||
| <template> | |||
| <router-view /> | |||
| </template> | |||
| <style lang="scss"> | |||
| html, | |||
| body { | |||
| margin: 0; | |||
| padding: 0; | |||
| height: 100%; | |||
| } | |||
| #app { | |||
| height: 100%; | |||
| } | |||
| // 链接样式 | |||
| .contributor-link { | |||
| color: #2196f3; // 默认使用适中的蓝色 | |||
| text-decoration: none; | |||
| transition: all 0.3s ease; // 添加过渡效果 | |||
| &:visited { | |||
| color: #2196f3; | |||
| } | |||
| &:hover { | |||
| color: #1976d2; // 悬停时使用深蓝色 | |||
| } | |||
| &:active { | |||
| color: #0d47a1 !important; // 点击时使用更深的蓝色 | |||
| } | |||
| } | |||
| </style> | |||
+ 34
- 0
src/AppMediaPipe.vue
View File
| @ -0,0 +1,34 @@ | |||
| <template> | |||
| <n-message-provider> | |||
| <div class="app-container"> | |||
| <h1>MediaPipe</h1> | |||
| <div class="detection-container"> | |||
| <hand-landmark-detection /> | |||
| </div> | |||
| </div> | |||
| </n-message-provider> | |||
| </template> | |||
| <script setup lang="ts"> | |||
| import HandLandmarkDetection from "@/hand_landmark/VideoDetector.vue"; | |||
| </script> | |||
| <style scoped> | |||
| .app-container { | |||
| padding: 20px; | |||
| display: flex; | |||
| flex-direction: column; | |||
| align-items: center; | |||
| } | |||
| h1 { | |||
| color: #333; | |||
| margin-bottom: 20px; | |||
| } | |||
| .detection-container { | |||
| width: 100%; | |||
| max-width: 640px; | |||
| margin: 0 auto; | |||
| } | |||
| </style> | |||
+ 31
- 0
src/components/AutoStart.vue
View File
| @ -0,0 +1,31 @@ | |||
| <script setup lang="ts"> | |||
| import { Power } from "@icon-park/vue-next"; | |||
| import { disable, enable } from "@tauri-apps/plugin-autostart"; | |||
| import { watch } from "vue"; | |||
| import { use_app_store } from "@/store/app"; | |||
| const app_store = use_app_store(); | |||
| watch( | |||
| () => app_store.config.auto_start, | |||
| async (value) => { | |||
| if (value) { | |||
| await enable(); | |||
| } else { | |||
| await disable(); | |||
| } | |||
| } | |||
| ); | |||
| </script> | |||
| <template> | |||
| <n-space align="center" style="display: flex; align-items: center"> | |||
| <span style="display: flex; align-items: center"> | |||
| <n-icon size="20" style="margin-right: 8px"> | |||
| <Power /> | |||
| </n-icon> | |||
| <span>{{ $t("开机自启动") }}</span> | |||
| </span> | |||
| <n-switch v-model:value="app_store.config.auto_start" /> | |||
| </n-space> | |||
| </template> | |||
+ 77
- 0
src/components/CircleProgress.vue
View File
| @ -0,0 +1,77 @@ | |||
| <template> | |||
| <div class="circle-progress"> | |||
| <svg :width="size" :height="size" viewBox="0 0 100 100"> | |||
| <!-- 背景圆环 --> | |||
| <circle | |||
| cx="50" | |||
| cy="50" | |||
| :r="radius" | |||
| fill="none" | |||
| :stroke="backgroundColor" | |||
| :stroke-width="strokeWidth" | |||
| /> | |||
| <!-- 进度圆环 --> | |||
| <circle | |||
| cx="50" | |||
| cy="50" | |||
| :r="radius" | |||
| fill="none" | |||
| :stroke="color" | |||
| :stroke-width="strokeWidth" | |||
| :stroke-dasharray="circumference" | |||
| :stroke-dashoffset="dashOffset" | |||
| class="progress" | |||
| /> | |||
| <!-- 中心文本 --> | |||
| <slot> | |||
| <text x="50" y="50" text-anchor="middle" dominant-baseline="middle" class="progress-text"> | |||
| {{ text }} | |||
| {{ percentage }}% | |||
| </text> | |||
| </slot> | |||
| </svg> | |||
| </div> | |||
| </template> | |||
| <script setup lang="ts"> | |||
| import { computed } from 'vue'; | |||
| interface Props { | |||
| percentage: number; | |||
| size?: number; | |||
| strokeWidth?: number; | |||
| color?: string; | |||
| backgroundColor?: string; | |||
| text?: string; | |||
| } | |||
| const props = withDefaults(defineProps<Props>(), { | |||
| size: 100, | |||
| strokeWidth: 6, | |||
| color: '#409eff', | |||
| backgroundColor: '#e5e9f2', | |||
| text: '' | |||
| }); | |||
| const radius = computed(() => 50 - props.strokeWidth / 2); | |||
| const circumference = computed(() => 2 * Math.PI * radius.value); | |||
| const dashOffset = computed(() => | |||
| circumference.value * (1 - props.percentage / 100) | |||
| ); | |||
| </script> | |||
| <style lang="scss" scoped> | |||
| .circle-progress { | |||
| display: inline-block; | |||
| .progress { | |||
| transform: rotate(-90deg); | |||
| transform-origin: center; | |||
| } | |||
| .progress-text { | |||
| font-size: 14px; | |||
| fill: #606266; | |||
| } | |||
| } | |||
| </style> | |||
+ 81
- 0
src/components/DevTool.vue
View File
| @ -0,0 +1,81 @@ | |||
| <template> | |||
| <div | |||
| v-if="is_dev" | |||
| class="dev-tool" | |||
| :class="{ 'dev-tool--expanded': isExpanded }" | |||
| > | |||
| <div class="dev-tool__toggle" @click="toggleToolbox"> | |||
| <span class="dev-tool__icon">🔧</span> | |||
| </div> | |||
| <!-- <div class="dev-tool__content" v-if="isExpanded"> | |||
| <div class="dev-tool__item" @click="createSubWindowClick">创建子窗口</div> | |||
| </div> --> | |||
| </div> | |||
| </template> | |||
| <script setup lang="ts"> | |||
| import { createSubWindow } from "@/utils/subWindow"; | |||
| import { ref } from "vue"; | |||
| const is_dev = import.meta.env.DEV; | |||
| const isExpanded = ref(false); | |||
| const toggleToolbox = () => { | |||
| isExpanded.value = !isExpanded.value; | |||
| }; | |||
| const createSubWindowClick = () => { | |||
| createSubWindow("/sub-window", "subWindow"); | |||
| }; | |||
| </script> | |||
| <style scoped lang="scss"> | |||
| .dev-tool { | |||
| position: fixed; | |||
| top: 50%; | |||
| left: 0; | |||
| transform: translateY(-50%); | |||
| background-color: #fff; | |||
| border-radius: 0 8px 8px 0; | |||
| box-shadow: 0 2px 8px rgba(0, 0, 0, 0.1); | |||
| transition: all 0.3s ease; | |||
| z-index: 9999; | |||
| &--expanded { | |||
| width: 200px; | |||
| } | |||
| &__toggle { | |||
| padding: 10px; | |||
| cursor: pointer; | |||
| display: flex; | |||
| align-items: center; | |||
| justify-content: center; | |||
| border-bottom: 1px solid #eee; | |||
| } | |||
| &__icon { | |||
| font-size: 20px; | |||
| } | |||
| &__content { | |||
| padding: 10px; | |||
| } | |||
| &__item { | |||
| padding: 8px 12px; | |||
| cursor: pointer; | |||
| border-radius: 4px; | |||
| margin-bottom: 5px; | |||
| transition: background-color 0.2s ease; | |||
| &:hover { | |||
| background-color: #f5f5f5; | |||
| } | |||
| &:last-child { | |||
| margin-bottom: 0; | |||
| } | |||
| } | |||
| } | |||
| </style> | |||
+ 82
- 0
src/components/GestureCard.vue
View File
| @ -0,0 +1,82 @@ | |||
| <template> | |||
| <n-card class="gesture-card" :bordered="false"> | |||
| <n-space align="center" class="gesture-content"> | |||
| <div class="gesture-icon" :class="{ 'double-hand': isDoubleHand }"> | |||
| <slot name="icon"></slot> | |||
| </div> | |||
| <div class="gesture-info"> | |||
| <h3>{{ title }}</h3> | |||
| <p>{{ description }}</p> | |||
| <slot name="extra"></slot> | |||
| </div> | |||
| </n-space> | |||
| </n-card> | |||
| </template> | |||
| <script setup lang="ts"> | |||
| defineProps<{ | |||
| title: string; | |||
| description: string; | |||
| isDoubleHand?: boolean; | |||
| }>(); | |||
| </script> | |||
| <style scoped lang="scss"> | |||
| .gesture-card { | |||
| transition: all 0.3s ease; | |||
| background: linear-gradient(145deg, #f8faff, #ffffff); | |||
| box-shadow: 0 4px 12px rgba(0, 0, 0, 0.05); | |||
| border: 1px solid #e5e9f2; | |||
| border-radius: 12px; | |||
| &:hover { | |||
| transform: translateY(-2px); | |||
| box-shadow: 0 8px 24px rgba(64, 152, 252, 0.15); | |||
| border-color: #4098fc; | |||
| } | |||
| } | |||
| .gesture-content { | |||
| padding: 12px; | |||
| } | |||
| .gesture-icon { | |||
| background: rgba(64, 152, 252, 0.1); | |||
| padding: 16px; | |||
| border-radius: 12px; | |||
| display: flex; | |||
| align-items: center; | |||
| justify-content: center; | |||
| border: 1px solid rgba(64, 152, 252, 0.2); | |||
| } | |||
| .gesture-info { | |||
| flex: 1; | |||
| h3 { | |||
| margin: 0 0 4px 0; | |||
| font-size: 1.1rem; | |||
| color: #2c3e50; | |||
| } | |||
| p { | |||
| margin: 0; | |||
| color: #666; | |||
| font-size: 0.9rem; | |||
| } | |||
| } | |||
| .double-hand { | |||
| display: flex; | |||
| gap: 8px; | |||
| :deep(svg) { | |||
| width: 32px; | |||
| height: 32px; | |||
| } | |||
| } | |||
| .flipped { | |||
| transform: scaleX(-1); | |||
| } | |||
| </style> | |||
+ 23
- 0
src/components/GestureIcon.vue
View File
| @ -0,0 +1,23 @@ | |||
| <template> | |||
| <component | |||
| :is="icon" | |||
| theme="outline" | |||
| size="40" | |||
| fill="#4098fc" | |||
| :stroke-width="3" | |||
| :class="{ flipped }" | |||
| /> | |||
| </template> | |||
| <script setup lang="ts"> | |||
| defineProps<{ | |||
| icon: any; | |||
| flipped?: boolean; | |||
| }>(); | |||
| </script> | |||
| <style scoped> | |||
| .flipped { | |||
| transform: scaleX(-1); | |||
| } | |||
| </style> | |||
+ 41
- 0
src/components/Menu.vue
View File
| @ -0,0 +1,41 @@ | |||
| <script setup lang="ts"> | |||
| import type { MenuOption } from "naive-ui"; | |||
| import { NMenu } from "naive-ui"; | |||
| import { ref } from "vue"; | |||
| import { useRouter } from "vue-router"; | |||
| const router = useRouter(); | |||
| const menuOptions: MenuOption[] = [ | |||
| { | |||
| label: "首页", | |||
| key: "/", | |||
| }, | |||
| { | |||
| label: "操作指南", | |||
| key: "/guide", | |||
| }, | |||
| // { | |||
| // label: "开发", | |||
| // key: "/update", | |||
| // } | |||
| ]; | |||
| const activeKey = ref("/"); | |||
| const handleUpdateValue = (key: string) => { | |||
| activeKey.value = key; | |||
| router.push(key); | |||
| }; | |||
| </script> | |||
| <template> | |||
| <n-menu | |||
| :options="menuOptions" | |||
| v-model:value="activeKey" | |||
| mode="vertical" | |||
| @update:value="handleUpdateValue" | |||
| /> | |||
| </template> | |||
| <style scoped></style> | |||
+ 259
- 0
src/hand_landmark/VideoDetector.vue
View File
| @ -0,0 +1,259 @@ | |||
| <template> | |||
| <div v-if="!camera_premission"> | |||
| <n-alert title="获取摄像头权限失败" type="error"> | |||
| <p>请尝试以下步骤解决:</p> | |||
| <ol> | |||
| <!-- <li> | |||
| 删除文件夹 | |||
| <n-tag size="small"> | |||
| %LOCALAPPDATA%\com.Lazyeat.maplelost\EBWebView | |||
| </n-tag> | |||
| </li> --> | |||
| <li> | |||
| 进入<n-tag size="small">%LOCALAPPDATA%\com.Lazyeat.maplelost</n-tag> | |||
| </li> | |||
| <li>删除<n-tag size="small">EBWebView</n-tag>文件夹</li> | |||
| <li>重新启动程序</li> | |||
| </ol> | |||
| </n-alert> | |||
| </div> | |||
| <div v-else> | |||
| <span>FPS: {{ FPS }}</span> | |||
| <div class="hand-detection"> | |||
| <video | |||
| ref="videoElement" | |||
| class="input-video" | |||
| :width="app_store.VIDEO_WIDTH" | |||
| :height="app_store.VIDEO_HEIGHT" | |||
| autoplay | |||
| style="display: none" | |||
| ></video> | |||
| <canvas | |||
| ref="canvasElement" | |||
| class="output-canvas" | |||
| :width="app_store.VIDEO_WIDTH" | |||
| :height="app_store.VIDEO_HEIGHT" | |||
| ></canvas> | |||
| </div> | |||
| <n-space> | |||
| <n-form-item label="识别框x"> | |||
| <n-input-number | |||
| v-model:value="app_store.config.boundary_left" | |||
| :min="0" | |||
| :max="app_store.VIDEO_WIDTH - app_store.config.boundary_width - 10" | |||
| style="width: 150px" | |||
| /> | |||
| </n-form-item> | |||
| <n-form-item label="识别框y"> | |||
| <n-input-number | |||
| v-model:value="app_store.config.boundary_top" | |||
| :min="0" | |||
| :max="app_store.VIDEO_HEIGHT - app_store.config.boundary_height - 10" | |||
| style="width: 150px" | |||
| /> | |||
| </n-form-item> | |||
| <n-form-item label="识别框宽"> | |||
| <n-input-number | |||
| v-model:value="app_store.config.boundary_width" | |||
| :min="0" | |||
| :max="app_store.VIDEO_WIDTH - app_store.config.boundary_left - 10" | |||
| style="width: 150px" | |||
| /> | |||
| </n-form-item> | |||
| <n-form-item label="识别框高"> | |||
| <n-input-number | |||
| v-model:value="app_store.config.boundary_height" | |||
| :min="0" | |||
| :max="app_store.VIDEO_HEIGHT - app_store.config.boundary_top - 10" | |||
| style="width: 150px" | |||
| /> | |||
| </n-form-item> | |||
| </n-space> | |||
| </div> | |||
| </template> | |||
| <script setup> | |||
| import { Detector } from "@/hand_landmark/detector"; | |||
| import { use_app_store } from "@/store/app"; | |||
| import { onBeforeUnmount, onMounted, ref, watch } from "vue"; | |||
| // 常量定义 | |||
| const app_store = use_app_store(); | |||
| // 组件状态 | |||
| const videoElement = ref(null); | |||
| const canvasElement = ref(null); | |||
| const detector = ref(new Detector()); | |||
| const lastVideoTime = ref(-1); | |||
| const currentStream = ref(null); | |||
| const FPS = ref(0); | |||
| const camera_premission = ref(false); | |||
| onMounted(() => { | |||
| navigator.mediaDevices.getUserMedia({ video: true }).then((stream) => { | |||
| camera_premission.value = true; | |||
| stream.getTracks().forEach((track) => track.stop()); | |||
| }); | |||
| }); | |||
| // 绘制相关方法 | |||
| const drawMouseMoveBox = (ctx) => { | |||
| ctx.strokeStyle = "rgb(255, 0, 255)"; | |||
| ctx.lineWidth = 2; | |||
| ctx.strokeRect( | |||
| app_store.config.boundary_left, | |||
| app_store.config.boundary_top, | |||
| app_store.config.boundary_width, | |||
| app_store.config.boundary_height | |||
| ); | |||
| }; | |||
| const frameCount = ref(0); | |||
| const fpsUpdateInterval = 1000; // 每秒更新一次 FPS | |||
| const lastFpsTime = ref(0); | |||
| const drawFPS = (ctx) => { | |||
| const now = performance.now(); | |||
| frameCount.value++; | |||
| if (now - lastFpsTime.value >= fpsUpdateInterval) { | |||
| FPS.value = frameCount.value; | |||
| frameCount.value = 0; | |||
| lastFpsTime.value = now; | |||
| } | |||
| }; | |||
| const drawHandLandmarks = (ctx, hand, color) => { | |||
| hand.landmarks.forEach((landmark) => { | |||
| ctx.beginPath(); | |||
| ctx.arc( | |||
| landmark.x * app_store.VIDEO_WIDTH, | |||
| landmark.y * app_store.VIDEO_HEIGHT, | |||
| 5, | |||
| 0, | |||
| 2 * Math.PI | |||
| ); | |||
| ctx.fillStyle = color; | |||
| ctx.fill(); | |||
| }); | |||
| }; | |||
| // 主要检测逻辑 | |||
| const predictWebcam = async () => { | |||
| const video = videoElement.value; | |||
| const canvas = canvasElement.value; | |||
| const ctx = canvas.getContext("2d"); | |||
| if (video.currentTime !== lastVideoTime.value) { | |||
| lastVideoTime.value = video.currentTime; | |||
| const detection = await detector.value.detect(video); | |||
| if (app_store.config.show_window) { | |||
| // 绘制视频帧 | |||
| ctx.clearRect(0, 0, canvas.width, canvas.height); | |||
| // 翻转绘制 | |||
| ctx.save(); | |||
| ctx.translate(canvas.width, 0); | |||
| ctx.scale(-1, 1); | |||
| ctx.drawImage(video, 0, 0, canvas.width, canvas.height); | |||
| // 绘制手势点 | |||
| if (detection.leftHand) { | |||
| drawHandLandmarks(ctx, detection.leftHand, "red"); | |||
| } | |||
| if (detection.rightHand) { | |||
| drawHandLandmarks(ctx, detection.rightHand, "blue"); | |||
| } | |||
| // 恢复绘制状态 | |||
| ctx.restore(); | |||
| // 绘制鼠标移动框 | |||
| drawMouseMoveBox(ctx); | |||
| } | |||
| // 手势处理 | |||
| await detector.value.process(detection); | |||
| // 绘制FPS | |||
| drawFPS(ctx); | |||
| } | |||
| requestAnimationFrame(predictWebcam); | |||
| }; | |||
| const initializeCamera = async () => { | |||
| try { | |||
| const stream = await navigator.mediaDevices.getUserMedia({ | |||
| video: { | |||
| deviceId: app_store.config.selected_camera_id | |||
| ? { exact: app_store.config.selected_camera_id } | |||
| : undefined, | |||
| width: app_store.VIDEO_WIDTH, | |||
| height: app_store.VIDEO_HEIGHT, | |||
| }, | |||
| audio: false, | |||
| }); | |||
| currentStream.value = stream; | |||
| videoElement.value.srcObject = stream; | |||
| videoElement.value.addEventListener("loadeddata", predictWebcam); | |||
| } catch (error) { | |||
| console.error("无法访问摄像头:", error); | |||
| } | |||
| }; | |||
| const stopCamera = () => { | |||
| if (videoElement.value?.srcObject) { | |||
| videoElement.value.srcObject.getTracks().forEach((track) => track.stop()); | |||
| } | |||
| }; | |||
| watch( | |||
| () => app_store.config.selected_camera_id, | |||
| async () => { | |||
| stopCamera(); | |||
| } | |||
| ); | |||
| // 监听 mission_running 的变化 | |||
| watch( | |||
| () => app_store.mission_running, | |||
| async (newValue) => { | |||
| if (newValue) { | |||
| await initializeCamera(); | |||
| app_store.flag_detecting = true; | |||
| } else { | |||
| stopCamera(); | |||
| } | |||
| } | |||
| ); | |||
| // 生命周期钩子 | |||
| onMounted(async () => { | |||
| await detector.value.initialize(); | |||
| // 如果 mission_running 为 true，则初始化摄像头 | |||
| if (app_store.mission_running) { | |||
| await initializeCamera(); | |||
| } | |||
| }); | |||
| onBeforeUnmount(() => { | |||
| stopCamera(); | |||
| }); | |||
| </script> | |||
| <style scoped> | |||
| .hand-detection { | |||
| width: v-bind('app_store.VIDEO_WIDTH + "px"'); | |||
| height: v-bind('app_store.VIDEO_HEIGHT + "px"'); | |||
| } | |||
| .output-canvas { | |||
| position: absolute; | |||
| } | |||
| </style> | |||
+ 291
- 0
src/hand_landmark/detector.ts
View File
| @ -0,0 +1,291 @@ | |||
| import { GestureHandler } from "@/hand_landmark/gesture_handler"; | |||
| import { | |||
| FilesetResolver, | |||
| GestureRecognizer, | |||
| GestureRecognizerOptions, | |||
| } from "@mediapipe/tasks-vision"; | |||
| // 手势枚举 | |||
| export enum HandGesture { | |||
| // 食指举起，移动鼠标 | |||
| ONLY_INDEX_UP = "only_index_up", | |||
| // 食指和拇指同时竖起 - 鼠标左键点击 | |||
| INDEX_AND_THUMB_UP = "index_and_thumb_up", | |||
| ROCK_GESTURE = "rock_gesture", | |||
| // 三根手指同时竖起 - 滚动屏幕 | |||
| THREE_FINGERS_UP = "three_fingers_up", | |||
| SCROLL_GESTURE_2 = "scroll_gesture_2", | |||
| // 四根手指同时竖起 | |||
| FOUR_FINGERS_UP = "four_fingers_up", | |||
| // 五根手指同时竖起 - 暂停/开始 识别 | |||
| STOP_GESTURE = "stop_gesture", | |||
| // 其他手势 | |||
| DELETE_GESTURE = "delete_gesture", | |||
| OTHER = "other", | |||
| } | |||
| interface HandLandmark { | |||
| x: number; | |||
| y: number; | |||
| z: number; | |||
| } | |||
| export interface HandInfo { | |||
| landmarks: HandLandmark[]; | |||
| handedness: "Left" | "Right"; | |||
| score: number; | |||
| categoryName?: string; | |||
| } | |||
| interface DetectionResult { | |||
| leftHand?: HandInfo; | |||
| rightHand?: HandInfo; | |||
| // 原始检测结果，以防需要访问其他数据 | |||
| rawResult: any; | |||
| } | |||
| /** | |||
| * 检测器类 - 负责手势识别和手势分类 | |||
| * 主要职责: | |||
| * 1. 初始化和管理MediaPipe HandLandmarker | |||
| * 2. 检测视频帧中的手部 | |||
| * 3. 分析手势类型(手指竖起等) | |||
| * 4. 提供手部关键点查询方法 | |||
| */ | |||
| export class Detector { | |||
| private detector: GestureRecognizer | null = null; | |||
| private gestureHandler: GestureHandler | null = null; | |||
| async initialize(useCanvas = false) { | |||
| const vision = await FilesetResolver.forVisionTasks("/mediapipe/wasm"); | |||
| try { | |||
| const params = { | |||
| baseOptions: { | |||
| modelAssetPath: "/mediapipe/gesture_recognizer.task", | |||
| delegate: "GPU", | |||
| }, | |||
| runningMode: "VIDEO", | |||
| numHands: 1, | |||
| } as GestureRecognizerOptions; | |||
| if (useCanvas) { | |||
| params.canvas = document.createElement("canvas"); | |||
| } | |||
| this.detector = await GestureRecognizer.createFromOptions(vision, params); | |||
| this.gestureHandler = new GestureHandler(); | |||
| } catch (error: any) { | |||
| // macos 旧设备的 wkwebview 对 webgl 兼容性不好，需要手动创建 canvas | |||
| if (error.toString().includes("kGpuService")) { | |||
| await this.initialize(true); | |||
| } else { | |||
| throw error; | |||
| } | |||
| } | |||
| } | |||
| /** | |||
| * 从视频帧检测手部 | |||
| */ | |||
| async detect(video: HTMLVideoElement): Promise<DetectionResult> { | |||
| if (!this.detector) { | |||
| throw new Error("检测器未初始化"); | |||
| } | |||
| const result = await this.detector.recognize(video); | |||
| const detection: DetectionResult = { | |||
| rawResult: result, | |||
| }; | |||
| if (result.landmarks && result.handedness) { | |||
| for (let i = 0; i < result.landmarks.length; i++) { | |||
| const hand: HandInfo = { | |||
| landmarks: result.landmarks[i], | |||
| handedness: result.handedness[i][0].categoryName as "Left" | "Right", | |||
| score: result.handedness[i][0].score, | |||
| }; | |||
| if (result.gestures.length > 0) { | |||
| hand.categoryName = result.gestures[0][0].categoryName; | |||
| } | |||
| if (hand.handedness === "Left") { | |||
| detection.leftHand = hand; | |||
| } else { | |||
| detection.rightHand = hand; | |||
| } | |||
| } | |||
| } | |||
| return detection; | |||
| } | |||
| /** | |||
| * 便捷方法：获取特定手指的关键点 | |||
| */ | |||
| static getFingerLandmarks( | |||
| hand: HandInfo | undefined, | |||
| fingerIndex: number | |||
| ): HandLandmark[] | null { | |||
| if (!hand) return null; | |||
| const fingerIndices = { | |||
| thumb: [1, 2, 3, 4], | |||
| index: [5, 6, 7, 8], | |||
| middle: [9, 10, 11, 12], | |||
| ring: [13, 14, 15, 16], | |||
| pinky: [17, 18, 19, 20], | |||
| }; | |||
| const indices = Object.values(fingerIndices)[fingerIndex]; | |||
| return indices.map((i) => hand.landmarks[i]); | |||
| } | |||
| /** | |||
| * 获取手指尖点 | |||
| */ | |||
| static getFingerTip( | |||
| hand: HandInfo | undefined, | |||
| fingerIndex: number | |||
| ): HandLandmark | null { | |||
| if (!hand) return null; | |||
| const tipIndices = [4, 8, 12, 16, 20]; | |||
| return hand.landmarks[tipIndices[fingerIndex]]; | |||
| } | |||
| /** | |||
| * 检测手指是否竖起 | |||
| */ | |||
| static _fingersUp(hand: HandInfo): number[] { | |||
| const fingers: number[] = []; | |||
| const tipIds = [4, 8, 12, 16, 20]; // 从大拇指开始，依次为每个手指指尖 | |||
| // 检测大拇指 | |||
| if (hand.handedness === "Right") { | |||
| if (hand.landmarks[tipIds[0]].x < hand.landmarks[tipIds[0] - 1].x) { | |||
| fingers.push(0); | |||
| } else { | |||
| fingers.push(1); | |||
| } | |||
| } else { | |||
| if (hand.landmarks[tipIds[0]].x > hand.landmarks[tipIds[0] - 1].x) { | |||
| fingers.push(0); | |||
| } else { | |||
| fingers.push(1); | |||
| } | |||
| } | |||
| // 检测其他四个手指 | |||
| for (let id = 1; id < 5; id++) { | |||
| if (hand.landmarks[tipIds[id]].y < hand.landmarks[tipIds[id] - 2].y) { | |||
| fingers.push(1); | |||
| } else { | |||
| fingers.push(0); | |||
| } | |||
| } | |||
| return fingers; | |||
| } | |||
| /** | |||
| * 获取单个手的手势类型 | |||
| */ | |||
| public static getSingleHandGesture(hand: HandInfo): HandGesture { | |||
| const fingers = this._fingersUp(hand); | |||
| const fingerState = fingers.join(","); | |||
| // 定义手势映射表 | |||
| const gestureMap = new Map<string, HandGesture>([ | |||
| // 食指举起，移动鼠标 | |||
| ["0,1,0,0,0", HandGesture.ONLY_INDEX_UP], | |||
| // 鼠标左键点击手势 | |||
| ["1,1,0,0,0", HandGesture.INDEX_AND_THUMB_UP], | |||
| ["0,1,0,0,1", HandGesture.ROCK_GESTURE], | |||
| ["1,1,0,0,1", HandGesture.ROCK_GESTURE], | |||
| // 滚动屏幕手势 | |||
| ["0,1,1,1,0", HandGesture.THREE_FINGERS_UP], | |||
| ["1,0,1,1,1", HandGesture.SCROLL_GESTURE_2], | |||
| ["0,0,1,1,1", HandGesture.SCROLL_GESTURE_2], | |||
| // 四根手指同时竖起 | |||
| ["0,1,1,1,1", HandGesture.FOUR_FINGERS_UP], | |||
| // 五根手指同时竖起 - 暂停/开始 识别 | |||
| ["1,1,1,1,1", HandGesture.STOP_GESTURE], | |||
| // 拇指和食指同时竖起 - 语音识别 | |||
| ["1,0,0,0,1", HandGesture.VOICE_GESTURE_START], | |||
| // 其他手势 | |||
| ["0,0,0,0,0", HandGesture.VOICE_GESTURE_STOP], | |||
| ]); | |||
| if (gestureMap.has(fingerState)) { | |||
| return gestureMap.get(fingerState) as HandGesture; | |||
| } | |||
| // 检查删除手势 | |||
| if (this._isDeleteGesture(hand, fingers)) { | |||
| return HandGesture.DELETE_GESTURE; | |||
| } | |||
| // 返回默认值 | |||
| return HandGesture.OTHER; | |||
| } | |||
| /** | |||
| * 检查是否为删除手势 | |||
| */ | |||
| private static _isDeleteGesture(hand: HandInfo, fingers: number[]): boolean { | |||
| const THUMB_INDEX = 4; | |||
| const FINGER_TIPS = [8, 12, 16, 20]; | |||
| const distance_threshold = 0.05; | |||
| const isThumbExtended = fingers[0] === 1; | |||
| const areOtherFingersClosed = fingers | |||
| .slice(1) | |||
| .every((finger) => finger === 0); | |||
| const isThumbLeftmost = FINGER_TIPS.every( | |||
| (tipIndex) => | |||
| hand.landmarks[THUMB_INDEX].x > | |||
| hand.landmarks[tipIndex].x + distance_threshold | |||
| ); | |||
| return isThumbExtended && areOtherFingersClosed && isThumbLeftmost; | |||
| } | |||
| /** | |||
| * 处理检测结果并执行相应动作 | |||
| */ | |||
| async process(detection: DetectionResult): Promise<void> { | |||
| const rightHandGesture = detection.rightHand | |||
| ? Detector.getSingleHandGesture(detection.rightHand) | |||
| : HandGesture.OTHER; | |||
| const leftHandGesture = detection.leftHand | |||
| ? Detector.getSingleHandGesture(detection.leftHand) | |||
| : HandGesture.OTHER; | |||
| // 优先使用右手 | |||
| let effectiveGesture = rightHandGesture; | |||
| if (detection.rightHand) { | |||
| effectiveGesture = rightHandGesture; | |||
| } else if (detection.leftHand) { | |||
| effectiveGesture = leftHandGesture; | |||
| } | |||
| // 将手势处理交给GestureHandler | |||
| if (detection.rightHand) { | |||
| this.gestureHandler?.handleGesture(effectiveGesture, detection.rightHand); | |||
| } else if (detection.leftHand) { | |||
| this.gestureHandler?.handleGesture(effectiveGesture, detection.leftHand); | |||
| } | |||
| } | |||
| } | |||
+ 507
- 0
src/hand_landmark/gesture_handler.ts
View File
| @ -0,0 +1,507 @@ | |||
| import { HandGesture, HandInfo } from "@/hand_landmark/detector"; | |||
| import i18n from "@/locales/i18n"; | |||
| import use_app_store from "@/store/app"; | |||
| // WebSocket数据类型定义 | |||
| enum WsDataType { | |||
| // 系统消息类型 | |||
| INFO = "info", | |||
| SUCCESS = "success", | |||
| WARNING = "warning", | |||
| ERROR = "error", | |||
| // 鼠标操作类型 | |||
| MOUSE_MOVE = "mouse_move", | |||
| MOUSE_CLICK = "mouse_click", | |||
| MOUSE_SCROLL_UP = "mouse_scroll_up", | |||
| MOUSE_SCROLL_DOWN = "mouse_scroll_down", | |||
| // 键盘操作类型 | |||
| SEND_KEYS = "send_keys", | |||
| // 语音操作类型 | |||
| VOICE_RECORD = "voice_record", | |||
| VOICE_STOP = "voice_stop", | |||
| } | |||
| interface WsData { | |||
| type: WsDataType; | |||
| msg?: string; | |||
| duration?: number; | |||
| title?: string; | |||
| data?: { | |||
| x?: number; | |||
| y?: number; | |||
| key_str?: string; | |||
| }; | |||
| } | |||
| /** | |||
| * 动作触发器类 - 负责发送操作命令到系统 | |||
| * 主要职责： | |||
| * 1. 维护WebSocket连接 | |||
| * 2. 提供各种操作方法（移动鼠标、点击等） | |||
| * 3. 发送通知 | |||
| */ | |||
| export class TriggerAction { | |||
| private ws: WebSocket | null = null; | |||
| constructor() { | |||
| this.connectWebSocket(); | |||
| } | |||
| private connectWebSocket() { | |||
| try { | |||
| this.ws = new WebSocket("ws://127.0.0.1:62334/ws_wavecontrol"); | |||
| this.ws.onmessage = (event: MessageEvent) => { | |||
| const response: WsData = JSON.parse(event.data); | |||
| const app_store = use_app_store(); | |||
| app_store.sub_window_info(response.msg || ""); | |||
| }; | |||
| this.ws.onopen = () => { | |||
| console.log("ws_wavecontrol connected"); | |||
| }; | |||
| this.ws.onclose = () => { | |||
| console.log("ws_wavecontrol closed, retrying..."); | |||
| this.ws = null; | |||
| setTimeout(() => this.connectWebSocket(), 3000); | |||
| }; | |||
| this.ws.onerror = (error) => { | |||
| console.error("ws_wavecontrol error:", error); | |||
| this.ws?.close(); | |||
| }; | |||
| } catch (error) { | |||
| console.error("Failed to create WebSocket instance:", error); | |||
| this.ws = null; | |||
| setTimeout(() => this.connectWebSocket(), 1000); | |||
| } | |||
| } | |||
| private send(data: { type: WsDataType } & Partial<Omit<WsData, "type">>) { | |||
| const message: WsData = { | |||
| type: data.type, | |||
| msg: data.msg || "", | |||
| title: data.title || "Lazyeat", | |||
| duration: data.duration || 1, | |||
| data: data.data || {}, | |||
| }; | |||
| this.ws?.send(JSON.stringify(message)); | |||
| } | |||
| moveMouse(x: number, y: number) { | |||
| this.send({ | |||
| type: WsDataType.MOUSE_MOVE, | |||
| data: { x, y }, | |||
| }); | |||
| } | |||
| clickMouse() { | |||
| this.send({ | |||
| type: WsDataType.MOUSE_CLICK, | |||
| }); | |||
| } | |||
| scrollUp() { | |||
| this.send({ | |||
| type: WsDataType.MOUSE_SCROLL_UP, | |||
| }); | |||
| } | |||
| scrollDown() { | |||
| this.send({ | |||
| type: WsDataType.MOUSE_SCROLL_DOWN, | |||
| }); | |||
| } | |||
| sendKeys(key_str: string) { | |||
| this.send({ | |||
| type: WsDataType.SEND_KEYS, | |||
| data: { key_str }, | |||
| }); | |||
| } | |||
| voiceRecord() { | |||
| this.send({ | |||
| type: WsDataType.VOICE_RECORD, | |||
| }); | |||
| } | |||
| voiceStop() { | |||
| this.send({ | |||
| type: WsDataType.VOICE_STOP, | |||
| }); | |||
| } | |||
| } | |||
| /** | |||
| * 手势处理器类 - 负责将手势转换为具体操作 | |||
| * 主要职责: | |||
| * 1. 接收识别到的手势类型 | |||
| * 2. 根据手势执行相应动作 | |||
| * 3. 处理防抖和连续手势确认 | |||
| */ | |||
| export class GestureHandler { | |||
| private triggerAction: TriggerAction; | |||
| private previousGesture: HandGesture | null = null; | |||
| private previousGestureCount: number = 0; | |||
| private minGestureCount: number = 5; | |||
| // 鼠标移动参数 | |||
| private screen_width: number = window.screen.width; | |||
| private screen_height: number = window.screen.height; | |||
| private smoothening = 7; // 平滑系数 | |||
| private prev_loc_x: number = 0; | |||
| private prev_loc_y: number = 0; | |||
| private prev_three_fingers_y: number = 0; // 添加三根手指上一次的 Y 坐标 | |||
| private prev_scroll2_y: number = 0; | |||
| // 时间间隔参数 | |||
| private lastClickTime: number = 0; | |||
| private lastScrollTime: number = 0; | |||
| private lastFullScreenTime: number = 0; | |||
| private lastDeleteTime: number = 0; | |||
| // 时间间隔常量（毫秒） | |||
| private readonly CLICK_INTERVAL = 500; // 点击间隔 | |||
| private readonly SCROLL_INTERVAL = 100; // 滚动间隔 | |||
| private readonly FULL_SCREEN_INTERVAL = 1500; // 全屏切换间隔 | |||
| // 语音识别参数 | |||
| private voice_recording: boolean = false; | |||
| private app_store: any; | |||
| constructor() { | |||
| this.triggerAction = new TriggerAction(); | |||
| this.app_store = use_app_store(); | |||
| } | |||
| /** | |||
| * 处理食指上举手势 - 鼠标移动 | |||
| */ | |||
| private handleIndexFingerUp(hand: HandInfo) { | |||
| const indexTip = this.getFingerTip(hand, 1); // 食指指尖 | |||
| if (!indexTip) return; | |||
| try { | |||
| // 将 hand 的坐标转换为视频坐标 | |||
| const video_x = | |||
| this.app_store.VIDEO_WIDTH - indexTip.x * this.app_store.VIDEO_WIDTH; | |||
| const video_y = indexTip.y * this.app_store.VIDEO_HEIGHT; | |||
| /** | |||
| * 辅助方法：将值从一个范围映射到另一个范围 | |||
| */ | |||
| function mapRange( | |||
| value: number, | |||
| fromMin: number, | |||
| fromMax: number, | |||
| toMin: number, | |||
| toMax: number | |||
| ): number { | |||
| return ( | |||
| ((value - fromMin) * (toMax - toMin)) / (fromMax - fromMin) + toMin | |||
| ); | |||
| } | |||
| // 将视频坐标映射到屏幕坐标 | |||
| // 由于 x 轴方向相反，所以需要翻转 | |||
| let screenX = mapRange( | |||
| video_x, | |||
| this.app_store.config.boundary_left, | |||
| this.app_store.config.boundary_left + | |||
| this.app_store.config.boundary_width, | |||
| 0, | |||
| this.screen_width | |||
| ); | |||
| let screenY = mapRange( | |||
| video_y, | |||
| this.app_store.config.boundary_top, | |||
| this.app_store.config.boundary_top + | |||
| this.app_store.config.boundary_height, | |||
| 0, | |||
| this.screen_height | |||
| ); | |||
| // 应用平滑处理 | |||
| screenX = | |||
| this.prev_loc_x + (screenX - this.prev_loc_x) / this.smoothening; | |||
| screenY = | |||
| this.prev_loc_y + (screenY - this.prev_loc_y) / this.smoothening; // 消除抖动 | |||
| this.prev_loc_x = screenX; | |||
| this.prev_loc_y = screenY; | |||
| // 移动鼠标 | |||
| this.app_store.sub_windows.x = screenX + 10; | |||
| this.app_store.sub_windows.y = screenY; | |||
| this.triggerAction.moveMouse(screenX, screenY); | |||
| } catch (error) { | |||
| console.error("处理鼠标移动失败:", error); | |||
| } | |||
| } | |||
| /** | |||
| * 处理食指和中指同时竖起手势 - 鼠标左键点击 | |||
| */ | |||
| private handleMouseClick() { | |||
| const now = Date.now(); | |||
| if (now - this.lastClickTime < this.CLICK_INTERVAL) { | |||
| return; | |||
| } | |||
| this.lastClickTime = now; | |||
| this.triggerAction.clickMouse(); | |||
| } | |||
| /** | |||
| * 处理三根手指同时竖起手势 - 滚动屏幕 | |||
| */ | |||
| private handleScroll(hand: HandInfo) { | |||
| const indexTip = this.getFingerTip(hand, 1); | |||
| const middleTip = this.getFingerTip(hand, 2); | |||
| const ringTip = this.getFingerTip(hand, 3); | |||
| if (!indexTip || !middleTip || !ringTip) { | |||
| this.prev_three_fingers_y = 0; | |||
| return; | |||
| } | |||
| const now = Date.now(); | |||
| if (now - this.lastScrollTime < this.SCROLL_INTERVAL) { | |||
| return; | |||
| } | |||
| this.lastScrollTime = now; | |||
| // 计算三根手指的平均 Y 坐标 | |||
| const currentY = (indexTip.y + middleTip.y + ringTip.y) / 3; | |||
| // 如果是第一次检测到手势，记录当前 Y 坐标 | |||
| if (this.prev_three_fingers_y === 0) { | |||
| this.prev_three_fingers_y = currentY; | |||
| return; | |||
| } | |||
| // 计算 Y 坐标的变化 | |||
| const deltaY = currentY - this.prev_three_fingers_y; | |||
| // 如果变化超过阈值，则触发滚动 | |||
| if (Math.abs(deltaY) > 0.008) { | |||
| if (deltaY < 0) { | |||
| // 手指向上移动，向上滚动 | |||
| this.triggerAction.scrollUp(); | |||
| } else { | |||
| // 手指向下移动，向下滚动 | |||
| this.triggerAction.scrollDown(); | |||
| } | |||
| // 更新上一次的 Y 坐标 | |||
| this.prev_three_fingers_y = currentY; | |||
| } | |||
| } | |||
| // 拇指和食指捏合，滚动屏幕 | |||
| private handleScroll2(hand: HandInfo) { | |||
| const indexTip = this.getFingerTip(hand, 1); | |||
| const thumbTip = this.getFingerTip(hand, 0); | |||
| if (!indexTip || !thumbTip) { | |||
| this.prev_scroll2_y = 0; | |||
| return; | |||
| } | |||
| const now = Date.now(); | |||
| if (now - this.lastScrollTime < this.SCROLL_INTERVAL) { | |||
| return; | |||
| } | |||
| this.lastScrollTime = now; | |||
| // 计算食指和拇指的距离 | |||
| const distance = Math.sqrt( | |||
| (indexTip.x - thumbTip.x) ** 2 + (indexTip.y - thumbTip.y) ** 2 | |||
| ); | |||
| console.log(i18n.global.t("当前食指和拇指距离"), distance); | |||
| // 如果距离大于阈值，说明没有捏合，重置上一次的 Y 坐标 | |||
| if ( | |||
| distance > | |||
| this.app_store.config.scroll_gesture_2_thumb_and_index_threshold | |||
| ) { | |||
| this.prev_scroll2_y = 0; | |||
| return; | |||
| } | |||
| // 如果是第一次检测到捏合，记录当前 Y 坐标 | |||
| if (this.prev_scroll2_y === 0) { | |||
| this.prev_scroll2_y = indexTip.y; | |||
| return; | |||
| } | |||
| // 计算 Y | |||
| const deltaY = indexTip.y - this.prev_scroll2_y; | |||
| // 如果变化超过阈值，则触发滚动 | |||
| if (Math.abs(deltaY) > 0.008) { | |||
| if (deltaY < 0) { | |||
| // 手指向上移动，向上滚动 | |||
| this.triggerAction.scrollUp(); | |||
| } else { | |||
| // 手指向下移动，向下滚动 | |||
| this.triggerAction.scrollDown(); | |||
| } | |||
| // 更新上一次的 Y 坐标 | |||
| this.prev_scroll2_y = indexTip.y; | |||
| } | |||
| } | |||
| /** | |||
| * 处理四根手指同时竖起手势 - 发送快捷键 | |||
| */ | |||
| private handleFourFingers() { | |||
| try { | |||
| const key_str = this.app_store.config.four_fingers_up_send || "f"; | |||
| const now = Date.now(); | |||
| if (now - this.lastFullScreenTime < this.FULL_SCREEN_INTERVAL) { | |||
| return; | |||
| } | |||
| this.lastFullScreenTime = now; | |||
| this.triggerAction.sendKeys(key_str); | |||
| } catch (error) { | |||
| console.error("处理四指手势失败:", error); | |||
| } | |||
| } | |||
| /** | |||
| * 处理拇指和小指同时竖起手势 - 开始语音识别 | |||
| */ | |||
| async handleVoiceStart() { | |||
| if (this.voice_recording) { | |||
| return; | |||
| } | |||
| await this.app_store.sub_window_info("开始语音识别"); | |||
| this.voice_recording = true; | |||
| this.triggerAction.voiceRecord(); | |||
| } | |||
| /** | |||
| * 处理拳头手势 - 停止语音识别 | |||
| */ | |||
| async handleVoiceStop() { | |||
| if (!this.voice_recording) { | |||
| return; | |||
| } | |||
| await this.app_store.sub_window_success("停止语音识别"); | |||
| this.voice_recording = false; | |||
| this.triggerAction.voiceStop(); | |||
| } | |||
| /** | |||
| * 处理删除手势 | |||
| */ | |||
| private handleDelete() { | |||
| const now = Date.now(); | |||
| if (now - this.lastDeleteTime < 300) { | |||
| return; | |||
| } | |||
| this.lastDeleteTime = now; | |||
| this.triggerAction.sendKeys("backspace"); | |||
| } | |||
| /** | |||
| * 处理停止手势 | |||
| */ | |||
| async handleStopGesture(): Promise<void> { | |||
| const toogle_detect = () => { | |||
| this.app_store.flag_detecting = !this.app_store.flag_detecting; | |||
| }; | |||
| if (this.previousGestureCount >= 60) { | |||
| toogle_detect(); | |||
| this.previousGestureCount = 0; | |||
| // 暂停手势识别后，更新 sub-window 进度条 | |||
| this.app_store.sub_windows.progress = 0; | |||
| } else { | |||
| this.previousGestureCount++; | |||
| if (this.previousGestureCount >= 20) { | |||
| // 更新 sub-window 进度条 | |||
| this.app_store.sub_windows.progress = Math.floor( | |||
| (this.previousGestureCount / 60) * 100 | |||
| ); | |||
| } | |||
| } | |||
| } | |||
| /** | |||
| * 获取手指尖点 | |||
| */ | |||
| private getFingerTip(hand: HandInfo, fingerIndex: number) { | |||
| if (!hand) return null; | |||
| const tipIndices = [4, 8, 12, 16, 20]; | |||
| return hand.landmarks[tipIndices[fingerIndex]]; | |||
| } | |||
| /** | |||
| * 处理手势 | |||
| */ | |||
| handleGesture(gesture: HandGesture, hand: HandInfo) { | |||
| // 更新手势连续性计数 | |||
| if (gesture === this.previousGesture) { | |||
| this.previousGestureCount++; | |||
| } else { | |||
| this.previousGesture = gesture; | |||
| this.previousGestureCount = 1; | |||
| } | |||
| // 首先处理停止手势 | |||
| if (gesture === HandGesture.STOP_GESTURE) { | |||
| if (hand.categoryName === "Open_Palm") { | |||
| this.handleStopGesture(); | |||
| } | |||
| return; | |||
| } | |||
| // 如果手势识别已暂停，则不处理 | |||
| if (!this.app_store.flag_detecting) { | |||
| return; | |||
| } | |||
| // 只要切换手势就停止语音识别 | |||
| if (gesture !== HandGesture.VOICE_GESTURE_START && this.voice_recording) { | |||
| this.handleVoiceStop(); | |||
| return; | |||
| } | |||
| // 鼠标移动手势直接执行，不需要连续确认 | |||
| if (gesture === HandGesture.ONLY_INDEX_UP) { | |||
| this.handleIndexFingerUp(hand); | |||
| return; | |||
| } | |||
| // 其他手势需要连续确认才执行 | |||
| if (this.previousGestureCount >= this.minGestureCount) { | |||
| switch (gesture) { | |||
| case HandGesture.ROCK_GESTURE: | |||
| case HandGesture.INDEX_AND_THUMB_UP: | |||
| this.handleMouseClick(); | |||
| break; | |||
| case HandGesture.SCROLL_GESTURE_2: | |||
| this.handleScroll2(hand); | |||
| break; | |||
| // case HandGesture.THREE_FINGERS_UP: | |||
| // this.handleScroll(hand); | |||
| // break; | |||
| case HandGesture.FOUR_FINGERS_UP: | |||
| this.handleFourFingers(); | |||
| break; | |||
| case HandGesture.VOICE_GESTURE_START: | |||
| this.handleVoiceStart(); | |||
| break; | |||
| case HandGesture.DELETE_GESTURE: | |||
| this.handleDelete(); | |||
| break; | |||
| } | |||
| } | |||
| } | |||
| } | |||
+ 49
- 0
src/locales/en.ts
View File
| @ -0,0 +1,49 @@ | |||
| export default { | |||
| "手势识别控制": "Gesture Recognition Control", | |||
| "运行中": "Running", | |||
| "已停止": "Stopped", | |||
| "开机自启动": "Auto Start", | |||
| "显示识别窗口": "Show Recognition Window", | |||
| "摄像头选择": "Camera Selection", | |||
| "手势操作指南": "Gesture Guide", | |||
| "光标控制": "Cursor Control", | |||
| "竖起食指滑动控制光标位置": "Slide with index finger to control cursor position", | |||
| "单击操作": "Click Operation", | |||
| "双指举起执行鼠标单击": "Raise two fingers to perform mouse click", | |||
| "Rock手势执行鼠标单击": "Rock gesture to perform mouse click", | |||
| "滚动控制": "Scroll Control", | |||
| "三指上下滑动控制页面滚动": "Slide three fingers up/down to control page scrolling", | |||
| "（okay手势）食指和拇指捏合滚动页面": "（okay gesture）Pinch with index and thumb to scroll page", | |||
| "食指和拇指距离小于": "Index and thumb distance less than", | |||
| "触发捏合": "Trigger pinch", | |||
| "默认值0.02": "Default value 0.02", | |||
| "可以通过右键->检查->控制台->捏合手势->查看当前距离": "Can check current distance by right-click -> inspect -> console -> pinch gesture", | |||
| "全屏控制": "Full Screen Control", | |||
| "四指并拢发送按键": "Four fingers together to send key", | |||
| "点击设置快捷键": "Click to set shortcut", | |||
| "请按下按键...": "Please press keys...", | |||
| "点击设置": "Click to set", | |||
| "退格": "Backspace", | |||
| "发送退格键": "Send backspace key", | |||
| "开始语音识别": "Start Voice Recognition", | |||
| "六指手势开始语音识别": "Six fingers gesture to start voice recognition", | |||
| "结束语音识别": "End Voice Recognition", | |||
| "拳头手势结束语音识别": "Fist gesture to end voice recognition", | |||
| "暂停/继续": "Pause/Resume", | |||
| "单手张开1.5秒 暂停/继续 手势识别": "Open one hand for 1.5 seconds to pause/resume gesture recognition", | |||
| "识别框x": "Recognition box x", | |||
| "识别框y": "Recognition box y", | |||
| "识别框宽": "Recognition box width", | |||
| "识别框高": "Recognition box height", | |||
| // 通知 | |||
| "Lazyeat": "Lazyeat", | |||
| "提示": "Tip", | |||
| "停止语音识别": "Stop Voice Recognition", | |||
| "手势识别": "Gesture Recognition", | |||
| "继续手势识别": "Continue Gesture Recognition", | |||
| "暂停手势识别": "Pause Gesture Recognition", | |||
| }; | |||
+ 18
- 0
src/locales/i18n.ts
View File
| @ -0,0 +1,18 @@ | |||
| import { createI18n } from "vue-i18n"; | |||
| // 导入语言包 | |||
| import en from "./en"; | |||
| import zh from "./zh"; | |||
| // 创建 i18n 实例 | |||
| const i18n = createI18n({ | |||
| locale: navigator.language.split("-")[0], // 使用系统语言作为默认语言 | |||
| // locale: "en", // 使用系统语言作为默认语言 | |||
| fallbackLocale: "zh", // 回退语言 | |||
| messages: { | |||
| en, | |||
| zh, | |||
| }, | |||
| }); | |||
| export default i18n; | |||
+ 47
- 0
src/locales/zh.ts
View File
| @ -0,0 +1,47 @@ | |||
| export default { | |||
| "手势识别控制": "手势识别控制", | |||
| "运行中": "运行中", | |||
| "已停止": "已停止", | |||
| "显示识别窗口": "显示识别窗口", | |||
| "摄像头选择": "摄像头选择", | |||
| "手势操作指南": "手势操作指南", | |||
| "开机自启动": "开机自启动", | |||
| "光标控制": "光标控制", | |||
| "竖起食指滑动控制光标位置": "竖起食指滑动控制光标位置", | |||
| "单击操作": "单击操作", | |||
| "双指举起执行鼠标单击": "双指举起执行鼠标单击", | |||
| "Rock手势执行鼠标单击": "Rock手势执行鼠标单击", | |||
| "滚动控制": "滚动控制", | |||
| "三指上下滑动控制页面滚动": "三指上下滑动控制页面滚动", | |||
| "（okay手势）食指和拇指捏合滚动页面": "（okay手势）食指和拇指捏合滚动页面", | |||
| "食指和拇指距离小于": "食指和拇指距离小于", | |||
| "触发捏合": "触发捏合", | |||
| "全屏控制": "全屏控制", | |||
| "四指并拢发送按键": "四指并拢发送按键", | |||
| "点击设置快捷键": "点击设置快捷键", | |||
| "请按下按键...": "请按下按键...", | |||
| "点击设置": "点击设置", | |||
| "退格": "退格", | |||
| "发送退格键": "发送退格键", | |||
| "开始语音识别": "开始语音识别", | |||
| "六指手势开始语音识别": "六指手势开始语音识别", | |||
| "结束语音识别": "结束语音识别", | |||
| "拳头手势结束语音识别": "拳头手势结束语音识别", | |||
| "暂停/继续": "暂停/继续", | |||
| "单手张开1.5秒 暂停/继续 手势识别": "单手张开1.5秒 暂停/继续 手势识别", | |||
| "识别框x": "识别框x", | |||
| "识别框y": "识别框y", | |||
| "识别框宽": "识别框宽", | |||
| "识别框高": "识别框高", | |||
| "默认值0.02": "默认值0.02", | |||
| "可以通过右键->检查->控制台->捏合手势->查看当前距离": "可以通过右键->检查->控制台->捏合手势->查看当前距离", | |||
| // 通知 | |||
| "Lazyeat": "Lazyeat", | |||
| "提示": "提示", | |||
| "停止语音识别": "停止语音识别", | |||
| "手势识别": "手势识别", | |||
| "继续手势识别": "继续手势识别", | |||
| "暂停手势识别": "暂停手势识别", | |||
| }; | |||
+ 85
- 0
src/main.ts
View File
| @ -0,0 +1,85 @@ | |||
| import i18n from "@/locales/i18n"; | |||
| import router from "@/router"; | |||
| import { createPinia } from "pinia"; | |||
| import { PiniaSharedState } from "pinia-shared-state"; | |||
| import { createApp } from "vue"; | |||
| import App from "@/App.vue"; | |||
| import { | |||
| create, | |||
| NAlert, | |||
| NButton, | |||
| NCard, | |||
| NCheckbox, | |||
| NDivider, | |||
| NForm, | |||
| NFormItem, | |||
| NIcon, | |||
| NImage, | |||
| NInput, | |||
| NLayout, | |||
| NLayoutContent, | |||
| NLayoutFooter, | |||
| NLayoutHeader, | |||
| NMenu, | |||
| NMessageProvider, | |||
| NSelect, | |||
| NSpace, | |||
| NInputNumber, | |||
| NSpin, | |||
| NSwitch, | |||
| NProgress, | |||
| NTag, | |||
| } from "naive-ui"; | |||
| // 引入element-plus | |||
| import "element-plus/dist/index.css"; | |||
| const naive = create({ | |||
| components: [ | |||
| NButton, | |||
| NLayout, | |||
| NLayoutHeader, | |||
| NLayoutContent, | |||
| NLayoutFooter, | |||
| NMenu, | |||
| NSpace, | |||
| NImage, | |||
| NDivider, | |||
| NSwitch, | |||
| NSelect, | |||
| NSpin, | |||
| NIcon, | |||
| NInput, | |||
| NInputNumber, | |||
| NForm, | |||
| NFormItem, | |||
| NCheckbox, | |||
| NCard, | |||
| NMessageProvider, | |||
| NAlert, | |||
| NTag, | |||
| NProgress, | |||
| ], | |||
| }); | |||
| const app = createApp(App); | |||
| const pinia = createPinia(); | |||
| // Pass the plugin to your application's pinia plugin | |||
| pinia.use( | |||
| PiniaSharedState({ | |||
| // Enables the plugin for all stores. Defaults to true. | |||
| enable: true, | |||
| // If set to true this tab tries to immediately recover the shared state from another tab. Defaults to true. | |||
| initialize: false, | |||
| // Enforce a type. One of native, idb, localstorage or node. Defaults to native. | |||
| type: "localstorage", | |||
| }) | |||
| ); | |||
| app.use(naive); | |||
| app.use(pinia); | |||
| app.use(i18n); | |||
| app.use(router); | |||
| app.mount("#app"); | |||
+ 30
- 0
src/py_api.ts
View File
| @ -0,0 +1,30 @@ | |||
| const port = 62334; | |||
| const base_url = `http://localhost:${port}`; | |||
| class PyApi { | |||
| async ready(): Promise<boolean> { | |||
| try { | |||
| await fetch(`${base_url}/`, { | |||
| signal: AbortSignal.timeout(1000), | |||
| }); | |||
| return true; | |||
| } catch (error) { | |||
| return false; | |||
| } | |||
| } | |||
| async shutdown() { | |||
| try { | |||
| await fetch(`${base_url}/shutdown`, { | |||
| method: "GET", | |||
| signal: AbortSignal.timeout(500), | |||
| }); | |||
| } catch (error) { | |||
| console.error("关闭服务失败:", error); | |||
| } | |||
| } | |||
| } | |||
| const pyApi = new PyApi(); | |||
| export default pyApi; | |||
+ 38
- 0
src/router/index.ts
View File
| @ -0,0 +1,38 @@ | |||
| import MainWindow from "@/view/mainWindow/MainWindow.vue"; | |||
| import SubWindow from "@/view/subWindow/SubWindow.vue"; | |||
| import Home from "@/view/mainWindow/Home.vue"; | |||
| import Update from "@/view/mainWindow/Update.vue"; | |||
| import Guide from "@/view/mainWindow/Guide.vue"; | |||
| import { createRouter, createWebHistory } from "vue-router"; | |||
| const routes = [ | |||
| { | |||
| path: "/", | |||
| name: "mainWindow", | |||
| component: MainWindow, | |||
| children: [ | |||
| { | |||
| path: "", | |||
| name: "home", | |||
| component: Home, | |||
| }, | |||
| { | |||
| path: "update", | |||
| name: "update", | |||
| component: Update, | |||
| }, | |||
| { | |||
| path: "guide", | |||
| name: "guide", | |||
| component: Guide, | |||
| } | |||
| ] | |||
| }, | |||
| ]; | |||
| const router = createRouter({ | |||
| history: createWebHistory(import.meta.env.BASE_URL), | |||
| routes, | |||
| }); | |||
| export default router; | |||
+ 87
- 0
src/store/app.ts
View File
| @ -0,0 +1,87 @@ | |||
| import { defineStore } from "pinia"; | |||
| interface Camera { | |||
| deviceId: string; | |||
| label: string; | |||
| kind: string; | |||
| } | |||
| enum NotiType { | |||
| INFO = "info", | |||
| SUCCESS = "success", | |||
| WARNING = "warning", | |||
| ERROR = "error", | |||
| } | |||
| export const use_app_store = defineStore("app-store", { | |||
| state: () => ({ | |||
| config: { | |||
| auto_start: false, | |||
| show_window: false, | |||
| four_fingers_up_send: "f", | |||
| selected_camera_id: "", | |||
| // 识别框 | |||
| boundary_left: 300, | |||
| boundary_top: 150, | |||
| boundary_width: 280, | |||
| boundary_height: 200, | |||
| // 手势识别 | |||
| scroll_gesture_2_thumb_and_index_threshold: 0.02, // 食指和拇指距离阈值 | |||
| }, | |||
| sub_windows: { | |||
| x: 0, | |||
| y: 0, | |||
| progress: 0, | |||
| notification: "", | |||
| noti_type: NotiType.INFO, | |||
| }, | |||
| mission_running: false, | |||
| cameras: [] as Camera[], | |||
| VIDEO_WIDTH: 640, | |||
| VIDEO_HEIGHT: 480, | |||
| flag_detecting: false, | |||
| }), | |||
| // PiniaSharedState 来共享不同 tauri 窗口之间的状态 | |||
| share: { | |||
| // Override global config for this store. | |||
| enable: true, | |||
| initialize: true, | |||
| }, | |||
| actions: { | |||
| is_macos() { | |||
| return navigator.userAgent.includes("Mac"); | |||
| }, | |||
| is_windows() { | |||
| return navigator.userAgent.includes("Windows"); | |||
| }, | |||
| is_linux() { | |||
| return navigator.userAgent.includes("Linux"); | |||
| }, | |||
| async sub_window_info(body: string) { | |||
| this.sub_windows.notification = body; | |||
| this.sub_windows.noti_type = NotiType.INFO; | |||
| }, | |||
| async sub_window_success(body: string) { | |||
| this.sub_windows.notification = body; | |||
| this.sub_windows.noti_type = NotiType.SUCCESS; | |||
| }, | |||
| async sub_window_warning(body: string) { | |||
| this.sub_windows.notification = body; | |||
| this.sub_windows.noti_type = NotiType.WARNING; | |||
| }, | |||
| async sub_window_error(body: string) { | |||
| this.sub_windows.notification = body; | |||
| this.sub_windows.noti_type = NotiType.ERROR; | |||
| }, | |||
| }, | |||
| }); | |||
| export default use_app_store; | |||
+ 46
- 0
src/utils/subWindow.ts
View File
| @ -0,0 +1,46 @@ | |||
| import { | |||
| WebviewWindow, | |||
| getAllWebviewWindows, | |||
| } from "@tauri-apps/api/webviewWindow"; | |||
| export const SUB_WINDOW_WIDTH = 130; | |||
| export const SUB_WINDOW_HEIGHT = 130; | |||
| export async function createSubWindow(url: string, title: string) { | |||
| let message = ""; | |||
| let success = true; | |||
| try { | |||
| const allWindows = await getAllWebviewWindows(); | |||
| const windownsLen = allWindows.length; | |||
| const label = `NewWindow_${windownsLen + 1}`; | |||
| const openUrl = url || "index.html"; | |||
| const newTitle = title || "新窗口"; | |||
| const openTitle = `${newTitle}-${windownsLen + 1}`; | |||
| const webview_window = new WebviewWindow(label, { | |||
| url: openUrl, | |||
| title: openTitle, | |||
| parent: "main", | |||
| zoomHotkeysEnabled: false, | |||
| width: SUB_WINDOW_WIDTH, | |||
| height: SUB_WINDOW_HEIGHT, | |||
| minWidth: SUB_WINDOW_WIDTH, | |||
| minHeight: SUB_WINDOW_HEIGHT, | |||
| alwaysOnTop: true, | |||
| decorations: false, // 隐藏窗口边框 | |||
| visible: false, | |||
| resizable: false, | |||
| }); | |||
| webview_window.once("tauri://created", async () => { | |||
| message = "打开成功"; | |||
| }); | |||
| webview_window.once("tauri://error", function (e) { | |||
| message = `打开${openTitle}报错: ${e}`; | |||
| success = false; | |||
| }); | |||
| return { success: success, message: message, webview: webview_window }; | |||
| } catch (error) { | |||
| return { success: false, message: error }; | |||
| } | |||
| } | |||
BIN
src/view/.DS_Store
View File
+ 348
- 0
src/view/mainWindow/Guide.vue
View File
| @ -0,0 +1,348 @@ | |||
| <template> | |||
| <div class="guide-container"> | |||
| <div class="header-bar"> | |||
| <div class="title-container"> | |||
| <h1 class="view-title">{{ $t('手势操作指南') }}</h1> | |||
| <div class="view-description">{{ $t('学习使用各种手势控制您的设备') }}</div> | |||
| </div> | |||
| </div> | |||
| <div class="main-content"> | |||
| <n-card class="guide-card"> | |||
| <div class="gesture-grid"> | |||
| <!-- 卡片1: 光标控制 --> | |||
| <div class="card"> | |||
| <div class="card-icon"> | |||
| <div class="icon-circle"> | |||
| <span class="icon-symbol">👆</span> | |||
| </div> | |||
| </div> | |||
| <div class="card-content"> | |||
| <h3 class="card-title">{{ $t('光标控制') }}</h3> | |||
| <p class="card-description">{{ $t('竖起食指滑动控制光标位置') }}</p> | |||
| </div> | |||
| </div> | |||
| <!-- 卡片2: 单击操作（双指） --> | |||
| <div class="card"> | |||
| <div class="card-icon"> | |||
| <div class="icon-circle"> | |||
| <span class="icon-symbol">✌️</span> | |||
| </div> | |||
| </div> | |||
| <div class="card-content"> | |||
| <h3 class="card-title">{{ $t('单击操作') }}</h3> | |||
| <p class="card-description">{{ $t('双指举起执行鼠标单击') }}</p> | |||
| </div> | |||
| </div> | |||
| <!-- 卡片3: 单击操作（Rock手势） --> | |||
| <div class="card"> | |||
| <div class="card-icon"> | |||
| <div class="icon-circle"> | |||
| <span class="icon-symbol">🤘</span> | |||
| </div> | |||
| </div> | |||
| <div class="card-content"> | |||
| <h3 class="card-title">{{ $t('单击操作') }}</h3> | |||
| <p class="card-description">{{ $t('Rock手势执行鼠标单击') }}</p> | |||
| <div class="card-extra"> | |||
| </div> | |||
| </div> | |||
| </div> | |||
| <!-- 卡片4: 滚动控制 --> | |||
| <div class="card"> | |||
| <div class="card-icon"> | |||
| <div class="icon-circle"> | |||
| <span class="icon-symbol">👌</span> | |||
| </div> | |||
| </div> | |||
| <div class="card-content"> | |||
| <h3 class="card-title">{{ $t('滚动控制') }}</h3> | |||
| <p class="card-description">{{ $t('(okay手势)食指和拇指捏合滚动页面') }}</p> | |||
| <div class="card-extra"> | |||
| <div class="setting-control"> | |||
| <span>{{ $t("食指和拇指距离小于") }}</span> | |||
| <n-input-number | |||
| v-model:value="app_store.config.scroll_gesture_2_thumb_and_index_threshold" | |||
| :min="0" | |||
| :step="0.01" | |||
| size="small" | |||
| /> | |||
| <span>{{ $t("触发捏合") }}</span> | |||
| </div> | |||
| <div class="hint-tags"> | |||
| <n-tag size="small" type="info">{{ $t("默认值0.02") }}</n-tag> | |||
| <n-tag size="small" type="info">{{ $t("右键->检查->控制台->查看当前距离") }}</n-tag> | |||
| </div> | |||
| </div> | |||
| </div> | |||
| </div> | |||
| <!-- 卡片5: 全屏控制 --> | |||
| <div class="card"> | |||
| <div class="card-icon"> | |||
| <div class="icon-circle"> | |||
| <span class="icon-symbol">🖐️</span> | |||
| </div> | |||
| </div> | |||
| <div class="card-content"> | |||
| <h3 class="card-title">{{ $t('全屏控制') }}</h3> | |||
| <p class="card-description">{{ $t('四指并拢发送按键') }}</p> | |||
| <div class="card-extra"> | |||
| <div class="keyboard-input"> | |||
| <n-input | |||
| :value="app_store.config.four_fingers_up_send || 'f'" | |||
| readonly | |||
| :placeholder="$t('点击设置快捷键')" | |||
| @click="listenForKey" | |||
| :status="isListening ? 'warning' : undefined" | |||
| size="small" | |||
| > | |||
| <template #suffix> | |||
| <span>{{ isListening ? $t("请按下按键...") : $t("点击设置") }}</span> | |||
| </template> | |||
| </n-input> | |||
| </div> | |||
| </div> | |||
| </div> | |||
| </div> | |||
| <!-- 卡片6: 退格 --> | |||
| <div class="card"> | |||
| <div class="card-icon"> | |||
| <div class="icon-circle"> | |||
| <span class="icon-symbol">🔙</span> | |||
| </div> | |||
| </div> | |||
| <div class="card-content"> | |||
| <h3 class="card-title">{{ $t('退格') }}</h3> | |||
| <p class="card-description">{{ $t('发送退格键') }}</p> | |||
| </div> | |||
| </div> | |||
| <!-- 卡片7: 开始语音识别 --> | |||
| <div class="card"> | |||
| <div class="card-icon"> | |||
| <div class="icon-circle"> | |||
| <span class="icon-symbol">🎤</span> | |||
| </div> | |||
| </div> | |||
| <div class="card-content"> | |||
| <h3 class="card-title">{{ $t('开始语音识别') }}</h3> | |||
| <p class="card-description">{{ $t('六指手势开始语音识别') }}</p> | |||
| </div> | |||
| </div> | |||
| <!-- 卡片8: 结束语音识别 --> | |||
| <div class="card"> | |||
| <div class="card-icon"> | |||
| <div class="icon-circle"> | |||
| <span class="icon-symbol">👊</span> | |||
| </div> | |||
| </div> | |||
| <div class="card-content"> | |||
| <h3 class="card-title">{{ $t('结束语音识别') }}</h3> | |||
| <p class="card-description">{{ $t('拳头手势结束语音识别') }}</p> | |||
| </div> | |||
| </div> | |||
| <!-- 卡片9: 暂停/继续 --> | |||
| <div class="card"> | |||
| <div class="card-icon"> | |||
| <div class="icon-circle"> | |||
| <span class="icon-symbol">✋</span> | |||
| </div> | |||
| </div> | |||
| <div class="card-content"> | |||
| <h3 class="card-title">{{ $t('暂停/继续') }}</h3> | |||
| <p class="card-description">{{ $t('单手张开1.5秒 暂停/继续 手势识别') }}</p> | |||
| </div> | |||
| </div> | |||
| </div> | |||
| </n-card> | |||
| </div> | |||
| </div> | |||
| </template> | |||
| <script setup lang="ts"> | |||
| import { use_app_store } from "@/store/app"; | |||
| import { ref } from "vue"; | |||
| const app_store = use_app_store(); | |||
| const isListening = ref(false); | |||
| const listenForKey = () => { | |||
| isListening.value = true; | |||
| // ...（原有监听键盘的代码） | |||
| }; | |||
| </script> | |||
| <style scoped lang="scss"> | |||
| .guide-container { | |||
| padding: 16px; | |||
| box-sizing: border-box; | |||
| } | |||
| .header-bar { | |||
| margin-bottom: 24px; | |||
| } | |||
| .title-container { | |||
| padding: 0 16px; | |||
| } | |||
| .view-title { | |||
| font-size: 24px; | |||
| font-weight: 700; | |||
| margin: 0; | |||
| color: #333; | |||
| } | |||
| .view-description { | |||
| font-size: 14px; | |||
| color: #666; | |||
| margin-top: 8px; | |||
| } | |||
| .main-content { | |||
| display: flex; | |||
| flex-direction: column; | |||
| } | |||
| .guide-card { | |||
| background-color: #fff; | |||
| border-radius: 8px; | |||
| box-shadow: 0 2px 8px rgba(0, 0, 0, 0.1); | |||
| overflow: hidden; | |||
| } | |||
| .gesture-grid { | |||
| display: grid; | |||
| grid-template-columns: repeat(3, 1fr); | |||
| gap: 16px; | |||
| padding: 16px; | |||
| } | |||
| .card { | |||
| display: flex; | |||
| flex-direction: column; | |||
| background: #fff; | |||
| border: 1px solid #f0f0f0; | |||
| border-radius: 8px; | |||
| overflow: hidden; | |||
| transition: all 0.3s ease; | |||
| &:hover { | |||
| box-shadow: 0 4px 12px rgba(0, 0, 0, 0.15); | |||
| transform: translateY(-2px); | |||
| } | |||
| } | |||
| .card-icon { | |||
| padding: 16px; | |||
| display: flex; | |||
| justify-content: center; | |||
| } | |||
| .icon-circle { | |||
| width: 64px; | |||
| height: 64px; | |||
| border-radius: 50%; | |||
| background: #e6f7ff; | |||
| display: flex; | |||
| align-items: center; | |||
| justify-content: center; | |||
| } | |||
| .icon-symbol { | |||
| font-size: 32px; | |||
| color: #1890ff; | |||
| } | |||
| .card-content { | |||
| padding: 0 16px 16px; | |||
| } | |||
| .card-title { | |||
| font-size: 16px; | |||
| font-weight: 600; | |||
| color: #333; | |||
| margin: 8px 0 4px; | |||
| } | |||
| .card-description { | |||
| font-size: 14px; | |||
| color: #666; | |||
| margin: 4px 0 8px; | |||
| line-height: 1.4; | |||
| } | |||
| .card-extra { | |||
| border-top: 1px solid #f0f0f0; | |||
| padding-top: 12px; | |||
| margin-top: 12px; | |||
| a { | |||
| color: #1890ff; | |||
| text-decoration: none; | |||
| margin-right: 12px; | |||
| &:hover { | |||
| text-decoration: underline; | |||
| } | |||
| } | |||
| } | |||
| .setting-control { | |||
| display: flex; | |||
| align-items: center; | |||
| flex-wrap: wrap; | |||
| gap: 8px; | |||
| margin-bottom: 8px; | |||
| .n-input-number { | |||
| width: 100px; | |||
| } | |||
| } | |||
| .hint-tags { | |||
| display: flex; | |||
| flex-wrap: wrap; | |||
| gap: 8px; | |||
| margin-top: 8px; | |||
| .n-tag { | |||
| padding: 4px 8px; | |||
| background: #f0f9ff; | |||
| border-color: #91d5ff; | |||
| color: #1890ff; | |||
| } | |||
| } | |||
| .keyboard-input { | |||
| width: 100%; | |||
| .n-input { | |||
| width: 100%; | |||
| } | |||
| } | |||
| @media (max-width: 992px) { | |||
| .gesture-grid { | |||
| grid-template-columns: repeat(2, 1fr); | |||
| } | |||
| } | |||
| @media (max-width: 576px) { | |||
| .gesture-grid { | |||
| grid-template-columns: 1fr; | |||
| } | |||
| .setting-control { | |||
| flex-direction: column; | |||
| align-items: flex-start; | |||
| } | |||
| } | |||
| </style> | |||
+ 367
- 0
src/view/mainWindow/Home.vue
View File
| @ -0,0 +1,367 @@ | |||
| <template> | |||
| <div class="home-container"> | |||
| <!-- 顶部统计数据卡片 --> | |||
| <div class="stats-row"> | |||
| <div class="stat-card"> | |||
| <div class="stat-icon">👋</div> | |||
| <div class="stat-data"> | |||
| <div class="value">92%</div> | |||
| <div class="label">识别准确率</div> | |||
| </div> | |||
| </div> | |||
| <div class="stat-card"> | |||
| <div class="stat-icon">⚡</div> | |||
| <div class="stat-data"> | |||
| <div class="value">38ms</div> | |||
| <div class="label">平均响应时间</div> | |||
| </div> | |||
| </div> | |||
| <div class="stat-card"> | |||
| <div class="stat-icon">🔍</div> | |||
| <div class="stat-data"> | |||
| <div class="value">312</div> | |||
| <div class="label">今日识别次数</div> | |||
| </div> | |||
| </div> | |||
| </div> | |||
| <!-- 主内容区域 --> | |||
| <div class="main-content"> | |||
| <!-- 左侧摄像头预览区域 --> | |||
| <div class="preview-card"> | |||
| <div class="preview-header"> | |||
| <h3>手势识别预览</h3> | |||
| <!-- 运行开关移到预览区域顶部 --> | |||
| <n-switch | |||
| v-model:value="app_store.mission_running" | |||
| size="large" | |||
| :rail-style="railStyle" | |||
| > | |||
| <template #checked>{{ $t("运行中") }}</template> | |||
| <template #unchecked>{{ $t("已停止") }}</template> | |||
| </n-switch> | |||
| </div> | |||
| <!-- 摄像头预览组件 --> | |||
| <VideoDetector /> | |||
| </div> | |||
| <!-- 右侧控制面板 --> | |||
| <div class="control-panel"> | |||
| <!-- 系统设置 --> | |||
| <div class="settings-section"> | |||
| <h3 class="section-title">手势识别控制</h3> | |||
| <div class="setting-item"> | |||
| <div class="setting-label"> | |||
| <n-icon size="20"> | |||
| <Browser /> | |||
| </n-icon> | |||
| <span>{{ $t("显示识别窗口") }}</span> | |||
| </div> | |||
| <n-switch v-model:value="app_store.config.show_window" /> | |||
| </div> | |||
| <div class="setting-item"> | |||
| <div class="setting-label"> | |||
| <AutoStart /> | |||
| </div> | |||
| </div> | |||
| <div class="setting-item"> | |||
| <div class="setting-label"> | |||
| <n-icon size="20"> | |||
| <Camera /> | |||
| </n-icon> | |||
| <span>{{ $t("摄像头选择") }}</span> | |||
| </div> | |||
| <n-select | |||
| v-model:value="app_store.config.selected_camera_id" | |||
| :options="camera_options" | |||
| :disabled="app_store.mission_running" | |||
| style="width: 100%" | |||
| /> | |||
| </div> | |||
| </div> | |||
| <!-- 手势库展示 --> | |||
| <div class="gesture-gallery"> | |||
| <div class="gallery-header"> | |||
| <div>手势库</div> | |||
| <button class="add-btn">+ 添加手势</button> | |||
| </div> | |||
| <div class="gesture-grid"> | |||
| <div class="gesture-item" v-for="i in 6" :key="i"> | |||
| <div class="gesture-icon">👌</div> | |||
| <div class="gesture-name">OK手势</div> | |||
| <div class="gesture-action">暂停媒体</div> | |||
| </div> | |||
| </div> | |||
| </div> | |||
| </div> | |||
| </div> | |||
| </div> | |||
| </template> | |||
| <script setup lang="ts"> | |||
| import AutoStart from "@/components/AutoStart.vue"; | |||
| import VideoDetector from "@/hand_landmark/VideoDetector.vue"; | |||
| import { use_app_store } from "@/store/app"; | |||
| import { Browser, Camera } from "@icon-park/vue-next"; | |||
| import { computed, onMounted } from "vue"; | |||
| const is_dev = computed(() => import.meta.env.DEV); | |||
| const app_store = use_app_store(); | |||
| // 计算属性：摄像头选项 | |||
| const camera_options = computed(() => { | |||
| return app_store.cameras.map((camera) => ({ | |||
| label: camera.label || `摄像头 ${camera.deviceId.slice(0, 4)}`, | |||
| value: camera.deviceId, | |||
| })); | |||
| }); | |||
| const getCameras = async () => { | |||
| try { | |||
| // 申请获取摄像头权限 | |||
| const stream = await navigator.mediaDevices.getUserMedia({ | |||
| video: true, | |||
| }); | |||
| stream.getTracks().forEach((track) => track.stop()); | |||
| const devices = await navigator.mediaDevices.enumerateDevices(); | |||
| app_store.cameras = devices.filter( | |||
| (device) => device.kind === "videoinput" | |||
| ); | |||
| } catch (error) { | |||
| console.error("获取摄像头列表失败:", error); | |||
| } | |||
| }; | |||
| onMounted(async () => { | |||
| await getCameras(); | |||
| }); | |||
| </script> | |||
| <style scoped lang="scss"> | |||
| .home-container { | |||
| display: flex; | |||
| flex-direction: column; | |||
| gap: 24px; | |||
| height: 100%; | |||
| } | |||
| .stats-row { | |||
| display: grid; | |||
| grid-template-columns: repeat(3, 1fr); | |||
| gap: 16px; | |||
| } | |||
| .stat-card { | |||
| background: var(--card-bg); | |||
| backdrop-filter: blur(10px); | |||
| border-radius: 16px; | |||
| padding: 20px; | |||
| display: flex; | |||
| align-items: center; | |||
| gap: 16px; | |||
| border: 1px solid var(--border-color); | |||
| transition: transform 0.2s, box-shadow 0.2s; | |||
| &:hover { | |||
| transform: translateY(-5px); | |||
| box-shadow: 0 10px 20px var(--shadow-color); | |||
| } | |||
| } | |||
| .stat-icon { | |||
| width: 56px; | |||
| height: 56px; | |||
| background: rgba(14, 165, 233, 0.1); | |||
| border-radius: 50%; | |||
| display: flex; | |||
| align-items: center; | |||
| justify-content: center; | |||
| font-size: 24px; | |||
| color: var(--accent-primary); | |||
| } | |||
| .stat-data { | |||
| flex: 1; | |||
| } | |||
| .value { | |||
| font-size: 24px; | |||
| font-weight: 700; | |||
| color: var(--text-primary); | |||
| } | |||
| .label { | |||
| font-size: 14px; | |||
| color: var(--text-secondary); | |||
| margin-top: 4px; | |||
| } | |||
| .main-content { | |||
| display: grid; | |||
| grid-template-columns: 1.5fr 1fr; | |||
| gap: 24px; | |||
| height: calc(100% - 120px); | |||
| } | |||
| .preview-card { | |||
| background: var(--card-bg); | |||
| backdrop-filter: blur(10px); | |||
| border-radius: 16px; | |||
| padding: 24px; | |||
| border: 1px solid var(--border-color); | |||
| display: flex; | |||
| flex-direction: column; | |||
| gap: 16px; | |||
| } | |||
| .preview-header { | |||
| display: flex; | |||
| justify-content: space-between; | |||
| align-items: center; | |||
| h3 { | |||
| font-size: 18px; | |||
| font-weight: 600; | |||
| color: var(--text-primary); | |||
| margin: 0; | |||
| } | |||
| } | |||
| .control-panel { | |||
| display: flex; | |||
| flex-direction: column; | |||
| gap: 24px; | |||
| } | |||
| .settings-section { | |||
| background: var(--card-bg); | |||
| backdrop-filter: blur(10px); | |||
| border-radius: 16px; | |||
| padding: 24px; | |||
| border: 1px solid var(--border-color); | |||
| } | |||
| .section-title { | |||
| font-size: 18px; | |||
| font-weight: 600; | |||
| color: var(--text-primary); | |||
| margin-top: 0; | |||
| margin-bottom: 20px; | |||
| } | |||
| .setting-item { | |||
| display: flex; | |||
| justify-content: space-between; | |||
| align-items: center; | |||
| padding: 12px 0; | |||
| border-bottom: 1px solid var(--border-color); | |||
| &:last-child { | |||
| border-bottom: none; | |||
| } | |||
| } | |||
| .setting-label { | |||
| display: flex; | |||
| align-items: center; | |||
| gap: 8px; | |||
| font-size: 14px; | |||
| color: var(--text-primary); | |||
| } | |||
| .gesture-gallery { | |||
| background: var(--card-bg); | |||
| backdrop-filter: blur(10px); | |||
| border-radius: 16px; | |||
| padding: 24px; | |||
| border: 1px solid var(--border-color); | |||
| } | |||
| .gallery-header { | |||
| display: flex; | |||
| justify-content: space-between; | |||
| align-items: center; | |||
| margin-bottom: 20px; | |||
| div { | |||
| font-size: 18px; | |||
| font-weight: 600; | |||
| color: var(--text-primary); | |||
| } | |||
| } | |||
| .add-btn { | |||
| background: rgba(14, 165, 233, 0.15); | |||
| border: none; | |||
| border-radius: 8px; | |||
| padding: 8px 15px; | |||
| color: var(--accent-primary); | |||
| font-weight: 500; | |||
| cursor: pointer; | |||
| transition: all 0.2s; | |||
| &:hover { | |||
| background: rgba(14, 165, 233, 0.25); | |||
| } | |||
| } | |||
| .gesture-grid { | |||
| display: grid; | |||
| grid-template-columns: repeat(2, 1fr); | |||
| gap: 15px; | |||
| margin-top: 10px; | |||
| } | |||
| .gesture-item { | |||
| background: var(--bg-tertiary); | |||
| border-radius: 12px; | |||
| padding: 15px; | |||
| display: flex; | |||
| flex-direction: column; | |||
| align-items: center; | |||
| transition: all 0.2s; | |||
| &:hover { | |||
| transform: translateY(-3px); | |||
| box-shadow: 0 4px 10px var(--shadow-color); | |||
| } | |||
| } | |||
| .gesture-icon { | |||
| font-size: 32px; | |||
| margin-bottom: 10px; | |||
| } | |||
| .gesture-name { | |||
| font-weight: 500; | |||
| color: var(--text-primary); | |||
| } | |||
| .gesture-action { | |||
| font-size: 13px; | |||
| color: var(--text-secondary); | |||
| text-align: center; | |||
| margin-top: 5px; | |||
| } | |||
| .ad-container { | |||
| height: 200px; | |||
| background: var(--card-bg); | |||
| backdrop-filter: blur(10px); | |||
| border-radius: 16px; | |||
| overflow: hidden; | |||
| iframe { | |||
| border: none; | |||
| background-color: var(--bg-secondary); | |||
| border-radius: 16px; | |||
| } | |||
| } | |||
| </style> | |||
+ 607
- 0
src/view/mainWindow/MainWindow.vue
View File
| @ -0,0 +1,607 @@ | |||
| <script setup lang="ts"> | |||
| import DevTool from "@/components/DevTool.vue"; | |||
| import pyApi from "@/py_api"; | |||
| import use_app_store from "@/store/app"; | |||
| import { getVersion } from "@tauri-apps/api/app"; | |||
| import { | |||
| getCurrentWindow, | |||
| LogicalPosition, | |||
| LogicalSize, | |||
| } from "@tauri-apps/api/window"; | |||
| import { LazyStore } from "@tauri-apps/plugin-store"; | |||
| import { onMounted, ref, watch, computed } from "vue"; | |||
| import { useRoute, useRouter } from "vue-router"; | |||
| const route = useRoute(); | |||
| const router = useRouter(); | |||
| const is_dev = import.meta.env.DEV; | |||
| const appVersion = ref(""); | |||
| const app_store = use_app_store(); | |||
| const ready = ref(false); | |||
| const darkTheme = ref(false); | |||
| const randomIconPath = ref(""); | |||
| // 新UI中需要的数据 | |||
| const settings = ref({ | |||
| autostart: true, | |||
| showWindow: true | |||
| }); | |||
| // 当前视图计算 | |||
| const currentView = computed(() => route.name || 'home'); | |||
| // 视图标题计算 | |||
| const viewTitle = computed(() => { | |||
| switch (currentView.value) { | |||
| case 'home': return '控制面板'; | |||
| case 'gestures': return '手势管理'; | |||
| case 'settings': return '设置中心'; | |||
| case 'help': return '帮助文档'; | |||
| default: return 'WaveControl'; | |||
| } | |||
| }); | |||
| // 在组件挂载时生成随机图标 | |||
| onMounted(() => { | |||
| const randomIndex = Math.floor(Math.random() * 9) + 1; | |||
| randomIconPath.value = `/wavecontrol${randomIndex}.jpg`; | |||
| }); | |||
| // 初始化逻辑 | |||
| onMounted(async () => { | |||
| ready.value = await pyApi.ready(); | |||
| const timer = setInterval(async () => { | |||
| ready.value = await pyApi.ready(); | |||
| if (ready.value) { | |||
| clearInterval(timer); | |||
| } | |||
| }, 5000); | |||
| await getCurrentWindow().onCloseRequested(async () => { | |||
| const factor = await getCurrentWindow().scaleFactor(); | |||
| const position = (await getCurrentWindow().innerPosition()).toLogical(factor); | |||
| const size = (await getCurrentWindow().innerSize()).toLogical(factor); | |||
| await window_store_json.set("window_state", { | |||
| x: position.x, | |||
| y: position.y, | |||
| width: size.width, | |||
| height: size.height, | |||
| }); | |||
| if (!is_dev) { | |||
| await pyApi.shutdown(); | |||
| } | |||
| }); | |||
| }); | |||
| // 窗口恢复上一次位置 | |||
| const window_store_json = new LazyStore("window_state.json"); | |||
| onMounted(async () => { | |||
| appVersion.value = await getVersion(); | |||
| const window_state = await window_store_json.get("window_state"); | |||
| if (window_state) { | |||
| let new_x = window_state.x; | |||
| let new_y = window_state.y; | |||
| const screen_width = window.screen.width; | |||
| const screen_height = window.screen.height; | |||
| // 如果窗口位置超出屏幕，则将窗口位置设置为100，100 | |||
| if (new_x <= 0) new_x = 100; | |||
| if (new_x >= screen_width) new_x = 100 | |||
| if (new_y <= 0) new_y = 100; | |||
| if (new_y >= screen_height) new_y = 100; | |||
| getCurrentWindow().setPosition(new LogicalPosition(new_x, new_y)); | |||
| getCurrentWindow().setSize( | |||
| new LogicalSize(window_state.width, window_state.height) | |||
| ); | |||
| } | |||
| }); | |||
| // app_store 数据加载 | |||
| const app_store_json = new LazyStore("settings.json"); | |||
| onMounted(async () => { | |||
| const config_data = await app_store_json.get("config"); | |||
| console.log("config_data", config_data); | |||
| if (config_data) { | |||
| Object.assign(app_store.config, JSON.parse(JSON.stringify(config_data))); | |||
| } | |||
| }); | |||
| watch( | |||
| () => app_store.config, | |||
| async (value) => { | |||
| await app_store_json.set("config", value); | |||
| app_store_json.save(); | |||
| }, | |||
| { deep: true } | |||
| ); | |||
| // 通知 | |||
| import { | |||
| isPermissionGranted, | |||
| requestPermission, | |||
| } from "@tauri-apps/plugin-notification"; | |||
| onMounted(async () => { | |||
| let permissionGranted = await isPermissionGranted(); | |||
| if (!permissionGranted) { | |||
| const permission = await requestPermission(); | |||
| permissionGranted = permission === "granted"; | |||
| } | |||
| }); | |||
| // 使用默认浏览器打开 iframe 中的 <a> 标签 | |||
| import { openUrl } from "@tauri-apps/plugin-opener"; | |||
| window.addEventListener("message", async function (e) { | |||
| const url = e.data; | |||
| if (url) { | |||
| await openUrl(url); | |||
| } | |||
| }); | |||
| // 创建子窗口 | |||
| import { createSubWindow } from "@/utils/subWindow"; | |||
| const subWindow = ref(null); | |||
| onMounted(async () => { | |||
| if (!subWindow.value) { | |||
| subWindow.value = await createSubWindow("/sub-window", "subWindow"); | |||
| } | |||
| }); | |||
| // 操作方法 | |||
| const toggleTheme = () => { | |||
| darkTheme.value = !darkTheme.value; | |||
| }; | |||
| const switchView = (viewName: string) => { | |||
| router.push({ name: viewName }); | |||
| }; | |||
| const toggleSetting = (setting: string) => { | |||
| settings.value[setting] = !settings.value[setting]; | |||
| }; | |||
| // 状态计算属性 | |||
| const statusClass = computed(() => ({ | |||
| 'status-active': ready.value, | |||
| 'status-inactive': !ready.value | |||
| })); | |||
| const statusText = computed(() => { | |||
| return ready.value ? '运行中' : '已停止'; | |||
| }); | |||
| </script> | |||
| <template> | |||
| <div class="app-container" :class="{'dark-theme': darkTheme}"> | |||
| <!-- 加载状态 --> | |||
| <div v-if="!ready" class="loading-overlay"> | |||
| <div class="loader-container"> | |||
| <div class="spinner"></div> | |||
| <div class="loading-text">手势识别模块加载中...</div> | |||
| </div> | |||
| </div> | |||
| <!-- 主界面 --> | |||
| <div v-else class="main-layout"> | |||
| <!-- 左侧导航 --> | |||
| <div class="sidebar"> | |||
| <div class="branding"> | |||
| <!-- 使用随机图标路径 --> | |||
| <img :src="randomIconPath" alt="WaveControl Logo" class="logo" /> | |||
| <div class="app-info"> | |||
| <div class="app-name">WaveControl</div> | |||
| </div> | |||
| </div> | |||
| <div class="nav-items"> | |||
| <div class="nav-item" :class="{active: currentView === 'home'}" @click="switchView('home')"> | |||
| <i class="icon">🏠</i> | |||
| <span>主控台</span> | |||
| </div> | |||
| <div class="nav-item" :class="{active: currentView === 'guide'}" @click="switchView('guide')"> | |||
| <i class="icon">👋</i> | |||
| <span>手势管理</span> | |||
| </div> | |||
| <div class="nav-item" :class="{active: currentView === 'settings'}" @click="switchView('settings')"> | |||
| <i class="icon">⚙️</i> | |||
| <span>设置</span> | |||
| </div> | |||
| <div class="nav-item" :class="{active: currentView === 'help'}" @click="switchView('help')"> | |||
| <i class="icon">❓</i> | |||
| <span>帮助</span> | |||
| </div> | |||
| </div> | |||
| <div class="status-card"> | |||
| <div class="status-indicator" :class="statusClass"> | |||
| {{ statusText }} | |||
| </div> | |||
| <div class="fps-counter"> | |||
| <span>FPS</span> | |||
| <span class="value">60</span> | |||
| </div> | |||
| </div> | |||
| <div class="contributors"> | |||
| <div v-if="app_store.is_macos()"> | |||
| <a | |||
| class="contributor-link" | |||
| href="https://gitee.com/wydhhh/software-engineering" | |||
| target="_blank" | |||
| > | |||
| @backpack</a | |||
| > | |||
| </div> | |||
| <div v-else class="contributor">@ninjakelly</div> | |||
| </div> | |||
| </div> | |||
| <!-- 主内容区 --> | |||
| <div class="content-area"> | |||
| <div class="header-bar"> | |||
| <div class="title-container"> | |||
| <h1 class="view-title">{{ viewTitle }}</h1> | |||
| <div class="view-description">隔空手势控制系统</div> | |||
| </div> | |||
| <div class="controls"> | |||
| <button class="theme-toggle" @click="toggleTheme"> | |||
| <i class="icon">{{ darkTheme ? '☀️' : '🌙' }}</i> | |||
| </button> | |||
| </div> | |||
| </div> | |||
| <div class="main-content"> | |||
| <router-view /> | |||
| </div> | |||
| </div> | |||
| </div> | |||
| <DevTool v-if="is_dev" /> | |||
| </div> | |||
| </template> | |||
| <style scoped lang="scss"> | |||
| .app-container { | |||
| font-family: 'Segoe UI', system-ui, sans-serif; | |||
| height: 100vh; | |||
| width: 100vw; | |||
| overflow: hidden; | |||
| transition: background-color 0.3s, color 0.3s; | |||
| &.dark-theme { | |||
| --bg-primary: #0f172a; | |||
| --bg-secondary: #1e293b; | |||
| --bg-tertiary: #334155; | |||
| --text-primary: #f1f5f9; | |||
| --text-secondary: #94a3b8; | |||
| --accent-primary: #0ea5e9; | |||
| --accent-secondary: #7dd3fc; | |||
| --hover-bg: rgba(148, 163, 184, 0.12); | |||
| --border-color: #334155; | |||
| --card-bg: rgba(30, 41, 59, 0.7); | |||
| --shadow-color: rgba(0, 0, 0, 0.4); | |||
| --status-active: #0ea5e9; | |||
| --status-inactive: #475569; | |||
| } | |||
| &:not(.dark-theme) { | |||
| --bg-primary: #f8fafc; | |||
| --bg-secondary: #f1f5f9; | |||
| --bg-tertiary: #e2e8f0; | |||
| --text-primary: #0f172a; | |||
| --text-secondary: #475569; | |||
| --accent-primary: #0284c7; | |||
| --accent-secondary: #0ea5e9; | |||
| --hover-bg: rgba(15, 23, 42, 0.05); | |||
| --border-color: #cbd5e1; | |||
| --card-bg: rgba(241, 245, 249, 0.7); | |||
| --shadow-color: rgba(0, 0, 0, 0.08); | |||
| --status-active: #0284c7; | |||
| --status-inactive: #94a3b8; | |||
| } | |||
| background-color: var(--bg-primary); | |||
| color: var(--text-primary); | |||
| } | |||
| .loading-overlay { | |||
| position: absolute; | |||
| top: 0; | |||
| left: 0; | |||
| width: 100%; | |||
| height: 100%; | |||
| display: flex; | |||
| align-items: center; | |||
| justify-content: center; | |||
| background-color: var(--bg-primary); | |||
| z-index: 100; | |||
| } | |||
| .loader-container { | |||
| display: flex; | |||
| flex-direction: column; | |||
| align-items: center; | |||
| gap: 20px; | |||
| } | |||
| .spinner { | |||
| width: 50px; | |||
| height: 50px; | |||
| border: 4px solid rgba(14, 165, 233, 0.2); | |||
| border-radius: 50%; | |||
| border-top: 4px solid var(--accent-primary); | |||
| animation: spin 1s linear infinite; | |||
| } | |||
| @keyframes spin { | |||
| 0% { transform: rotate(0deg); } | |||
| 100% { transform: rotate(360deg); } | |||
| } | |||
| .loading-text { | |||
| font-size: 18px; | |||
| font-weight: 500; | |||
| color: var(--text-primary); | |||
| } | |||
| .main-layout { | |||
| display: flex; | |||
| height: 100vh; | |||
| } | |||
| /* 左侧导航样式 */ | |||
| .sidebar { | |||
| width: 260px; | |||
| background-color: var(--bg-secondary); | |||
| display: flex; | |||
| flex-direction: column; | |||
| padding: 25px 0; | |||
| border-right: 1px solid var(--border-color); | |||
| z-index: 10; | |||
| } | |||
| .branding { | |||
| display: flex; | |||
| align-items: center; | |||
| padding: 0 20px 20px; | |||
| } | |||
| .logo { | |||
| width: 42px; | |||
| height: 42px; | |||
| border-radius: 10px; | |||
| margin-right: 12px; | |||
| background: linear-gradient(135deg, var(--accent-primary), #7dd3fc); | |||
| padding: 5px; | |||
| } | |||
| .app-info { | |||
| display: flex; | |||
| flex-direction: column; | |||
| } | |||
| .app-name { | |||
| font-size: 18px; | |||
| font-weight: 700; | |||
| letter-spacing: 0.5px; | |||
| color: var(--text-primary); | |||
| } | |||
| .app-version { | |||
| font-size: 13px; | |||
| color: var(--text-secondary); | |||
| font-weight: 500; | |||
| opacity: 0.8; | |||
| } | |||
| .nav-items { | |||
| display: flex; | |||
| flex-direction: column; | |||
| gap: 8px; | |||
| padding: 0 15px; | |||
| flex: 1; | |||
| } | |||
| .nav-item { | |||
| display: flex; | |||
| align-items: center; | |||
| padding: 12px 20px; | |||
| border-radius: 10px; | |||
| cursor: pointer; | |||
| transition: all 0.2s; | |||
| gap: 14px; | |||
| font-size: 16px; | |||
| color: var(--text-secondary); | |||
| &:hover { | |||
| background-color: var(--hover-bg); | |||
| color: var(--text-primary); | |||
| } | |||
| &.active { | |||
| background: rgba(14, 165, 233, 0.15); | |||
| color: var(--accent-primary); | |||
| font-weight: 500; | |||
| .icon { | |||
| color: var(--accent-primary); | |||
| } | |||
| } | |||
| } | |||
| .icon { | |||
| font-size: 18px; | |||
| } | |||
| .status-card { | |||
| background: var(--card-bg); | |||
| border-radius: 12px; | |||
| padding: 20px; | |||
| margin: 20px; | |||
| backdrop-filter: blur(10px); | |||
| border: 1px solid var(--border-color); | |||
| box-shadow: 0 4px 6px var(--shadow-color); | |||
| } | |||
| .status-indicator { | |||
| display: inline-flex; | |||
| align-items: center; | |||
| font-size: 14px; | |||
| font-weight: 500; | |||
| padding: 6px 12px; | |||
| border-radius: 20px; | |||
| &::before { | |||
| content: ''; | |||
| display: inline-block; | |||
| width: 8px; | |||
| height: 8px; | |||
| border-radius: 50%; | |||
| margin-right: 8px; | |||
| } | |||
| &.status-active { | |||
| background: rgba(14, 165, 233, 0.2); | |||
| color: var(--accent-primary); | |||
| &::before { | |||
| background-color: var(--accent-primary); | |||
| } | |||
| } | |||
| &.status-inactive { | |||
| background: rgba(148, 163, 184, 0.2); | |||
| color: var(--text-secondary); | |||
| &::before { | |||
| background-color: var(--text-secondary); | |||
| } | |||
| } | |||
| } | |||
| .fps-counter { | |||
| margin-top: 15px; | |||
| display: flex; | |||
| align-items: baseline; | |||
| gap: 8px; | |||
| span { | |||
| font-size: 14px; | |||
| color: var(--text-secondary); | |||
| } | |||
| .value { | |||
| font-size: 24px; | |||
| font-weight: 600; | |||
| color: var(--accent-primary); | |||
| } | |||
| } | |||
| .contributors { | |||
| padding: 20px; | |||
| color: var(--text-secondary); | |||
| font-size: 14px; | |||
| display: flex; | |||
| flex-direction: column; | |||
| gap: 15px; | |||
| .contributor { | |||
| padding: 5px 12px; | |||
| border-radius: 6px; | |||
| background-color: var(--bg-tertiary); | |||
| } | |||
| .contributor-link { | |||
| color: var(--accent-primary); | |||
| text-decoration: none; | |||
| &:hover { | |||
| text-decoration: underline; | |||
| } | |||
| } | |||
| } | |||
| /* 主内容区域样式 */ | |||
| .content-area { | |||
| flex: 1; | |||
| display: flex; | |||
| flex-direction: column; | |||
| overflow: hidden; | |||
| } | |||
| .header-bar { | |||
| display: flex; | |||
| justify-content: space-between; | |||
| align-items: center; | |||
| padding: 25px 30px 15px; | |||
| border-bottom: 1px solid var(--border-color); | |||
| } | |||
| .title-container { | |||
| display: flex; | |||
| flex-direction: column; | |||
| } | |||
| .view-title { | |||
| font-size: 28px; | |||
| font-weight: 700; | |||
| margin: 0; | |||
| color: var(--text-primary); | |||
| line-height: 1.2; | |||
| } | |||
| .view-description { | |||
| font-size: 15px; | |||
| color: var(--text-secondary); | |||
| margin-top: 6px; | |||
| } | |||
| .controls { | |||
| display: flex; | |||
| gap: 15px; | |||
| } | |||
| .theme-toggle { | |||
| width: 40px; | |||
| height: 40px; | |||
| border-radius: 50%; | |||
| background: var(--bg-secondary); | |||
| border: none; | |||
| display: flex; | |||
| align-items: center; | |||
| justify-content: center; | |||
| cursor: pointer; | |||
| font-size: 18px; | |||
| color: var(--text-primary); | |||
| transition: all 0.2s; | |||
| &:hover { | |||
| background: var(--hover-bg); | |||
| transform: scale(1.05); | |||
| } | |||
| } | |||
| .main-content { | |||
| flex: 1; | |||
| overflow: auto; | |||
| padding: 30px; | |||
| } | |||
| /* 添加路由视图过渡效果 */ | |||
| .router-view-container { | |||
| transition: opacity 0.3s ease; | |||
| > * { | |||
| animation: fadeIn 0.5s ease; | |||
| } | |||
| } | |||
| @keyframes fadeIn { | |||
| from { opacity: 0; transform: translateY(10px); } | |||
| to { opacity: 1; transform: translateY(0); } | |||
| } | |||
| </style> | |||
+ 95
- 0
src/view/mainWindow/Update.vue
View File
| @ -0,0 +1,95 @@ | |||
| <template> | |||
| <div class="update-container"> | |||
| <n-card title="开发" hoverable> | |||
| <n-space vertical size="large"> | |||
| <div class="dev-section"> | |||
| <n-alert type="info" title="开发者招募"> | |||
| <template #icon> | |||
| <n-icon><code /></n-icon> | |||
| </template> | |||
| <n-space vertical> | |||
| <span>有安卓开发的大佬吗，可以联系我。</span> | |||
| <span>欢迎加入QQ群 | |||
| <n-button text type="primary" tag="a" href="https://jq.qq.com/?_wv=1027&k=452246065" target="_blank"> | |||
| 452246065 | |||
| </n-button> | |||
| 讨论。 | |||
| </span> | |||
| </n-space> | |||
| </n-alert> | |||
| </div> | |||
| <div class="version-section"> | |||
| <h3>最新版本</h3> | |||
| <n-card embedded> | |||
| <n-space vertical> | |||
| <div class="version-header"> | |||
| <n-tag type="success" size="small">v0.3.9</n-tag> | |||
| <n-button text type="primary" tag="a" href="https://github.com/maplelost/lazyeat/releases/tag/v0.3.9" target="_blank"> | |||
| 下载地址 | |||
| </n-button> | |||
| </div> | |||
| <n-list> | |||
| <n-list-item> | |||
| <n-text type="success">新增</n-text> 捏合手势滚动功能 | |||
| </n-list-item> | |||
| <n-list-item> | |||
| <n-text type="info">优化</n-text> 自定义手势识别区域 | |||
| </n-list-item> | |||
| <n-list-item> | |||
| <n-text type="info">优化</n-text> 手势识别的准确性 | |||
| </n-list-item> | |||
| </n-list> | |||
| </n-space> | |||
| </n-card> | |||
| </div> | |||
| </n-space> | |||
| </n-card> | |||
| </div> | |||
| </template> | |||
| <script setup lang="ts"> | |||
| import { NCard, NSpace, NList, NListItem, NAlert, NButton, NTag, NText, NIcon } from 'naive-ui' | |||
| import { Code } from '@icon-park/vue-next' | |||
| </script> | |||
| <style scoped> | |||
| .update-container { | |||
| padding: 16px; | |||
| max-width: 800px; | |||
| margin: 0 auto; | |||
| } | |||
| .dev-section { | |||
| margin-bottom: 24px; | |||
| } | |||
| .version-section { | |||
| margin-bottom: 24px; | |||
| } | |||
| .version-section h3 { | |||
| margin: 0 0 16px 0; | |||
| color: #2c3e50; | |||
| font-size: 1.2em; | |||
| } | |||
| .version-header { | |||
| display: flex; | |||
| align-items: center; | |||
| gap: 12px; | |||
| margin-bottom: 16px; | |||
| } | |||
| :deep(.n-alert .n-alert-body) { | |||
| align-items: flex-start; | |||
| } | |||
| :deep(.n-card) { | |||
| transition: all 0.3s ease; | |||
| } | |||
| :deep(.n-card:hover) { | |||
| box-shadow: 0 4px 12px rgba(0, 0, 0, 0.1); | |||
| } | |||
| </style> | |||
+ 93
- 0
src/view/subWindow/SubWindow.vue
View File
| @ -0,0 +1,93 @@ | |||
| <template> | |||
| <div class="container-sub-window"> | |||
| <CircleProgress | |||
| :percentage="app_store.sub_windows.progress" | |||
| :size="100" | |||
| :text="app_store.flag_detecting ? '暂停检测' : '继续检测'" | |||
| :color="app_store.flag_detecting ? '#F56C6C' : '#67C23A'" | |||
| /> | |||
| <div style="height: 30px"> | |||
| <span>{{ app_store.sub_windows.notification }}</span> | |||
| </div> | |||
| </div> | |||
| </template> | |||
| <script setup lang="ts"> | |||
| import CircleProgress from "@/components/CircleProgress.vue"; | |||
| import use_app_store from "@/store/app"; | |||
| import { getCurrentWindow, LogicalPosition } from "@tauri-apps/api/window"; | |||
| import { computed, ref, watch } from "vue"; | |||
| const app_store = use_app_store(); | |||
| const display_progress = ref(false); | |||
| const display_notification = computed(() => { | |||
| return !display_progress.value; | |||
| }); | |||
| let hideTimer: number | null = null; | |||
| async function show_window() { | |||
| await getCurrentWindow().show(); | |||
| } | |||
| async function hide_window() { | |||
| await getCurrentWindow().hide(); | |||
| } | |||
| watch( | |||
| () => app_store.sub_windows.x, | |||
| (newVal) => { | |||
| getCurrentWindow().setPosition( | |||
| new LogicalPosition(newVal, app_store.sub_windows.y) | |||
| ); | |||
| } | |||
| ); | |||
| // 显示 sub-window | |||
| watch( | |||
| () => app_store.sub_windows.progress, | |||
| (newVal) => { | |||
| if (newVal) { | |||
| display_progress.value = true; | |||
| show_window(); | |||
| // 清除之前的定时器 | |||
| if (hideTimer) { | |||
| clearTimeout(hideTimer); | |||
| } | |||
| // 设置新的定时器 | |||
| hideTimer = setTimeout(() => { | |||
| hide_window(); | |||
| }, 300); | |||
| } | |||
| } | |||
| ); | |||
| watch( | |||
| () => app_store.sub_windows.notification, | |||
| (newVal) => { | |||
| if (newVal) { | |||
| display_progress.value = false; | |||
| show_window(); | |||
| // 清除之前的定时器 | |||
| if (hideTimer) { | |||
| clearTimeout(hideTimer); | |||
| } | |||
| // 设置新的定时器 | |||
| hideTimer = setTimeout(() => { | |||
| hide_window(); | |||
| app_store.sub_windows.notification = ""; | |||
| }, 1000); | |||
| } | |||
| } | |||
| ); | |||
| </script> | |||
| <style lang="scss" scoped> | |||
| .container-sub-window { | |||
| width: 100%; | |||
| height: 100%; | |||
| display: flex; | |||
| justify-content: center; | |||
| align-items: center; | |||
| flex-direction: column; | |||
| } | |||
| </style> | |||
+ 7
- 0
src/vite-env.d.ts
View File
| @ -0,0 +1,7 @@ | |||
| /// <reference types="vite/client" /> | |||
| declare module "*.vue" { | |||
| import type { DefineComponent } from "vue"; | |||
| const component: DefineComponent<{}, {}, any>; | |||
| export default component; | |||
| } | |||
+ 30
- 0
tsconfig.json
View File
| @ -0,0 +1,30 @@ | |||
| { | |||
| "compilerOptions": { | |||
| "target": "ES2020", | |||
| "useDefineForClassFields": true, | |||
| "module": "ESNext", | |||
| "lib": ["ES2020", "DOM", "DOM.Iterable"], | |||
| "skipLibCheck": true, | |||
| /* Bundler mode */ | |||
| "moduleResolution": "bundler", | |||
| "allowImportingTsExtensions": true, | |||
| "resolveJsonModule": true, | |||
| "isolatedModules": true, | |||
| "noEmit": true, | |||
| "jsx": "preserve", | |||
| /* Linting */ | |||
| "strict": true, | |||
| "noUnusedLocals": true, | |||
| "noUnusedParameters": true, | |||
| "noFallthroughCasesInSwitch": true, | |||
| "baseUrl": ".", | |||
| "paths": { | |||
| "@/*": ["src/*"] // 将 @ 映射到 src 目录 | |||
| } | |||
| }, | |||
| "include": ["src/**/*.ts", "src/**/*.d.ts", "src/**/*.tsx", "src/**/*.vue"], | |||
| "references": [{ "path": "./tsconfig.node.json" }] | |||
| } | |||
+ 10
- 0
tsconfig.node.json
View File
| @ -0,0 +1,10 @@ | |||
| { | |||
| "compilerOptions": { | |||
| "composite": true, | |||
| "skipLibCheck": true, | |||
| "module": "ESNext", | |||
| "moduleResolution": "bundler", | |||
| "allowSyntheticDefaultImports": true | |||
| }, | |||
| "include": ["vite.config.ts"] | |||
| } | |||
+ 43
- 0
vite.config.ts
View File
| @ -0,0 +1,43 @@ | |||
| import { defineConfig } from "vite"; | |||
| import vue from "@vitejs/plugin-vue"; | |||
| import path from "path"; | |||
| import { fileURLToPath } from "url"; | |||
| import { dirname } from "path"; | |||
| // @ts-expect-error process is a nodejs global | |||
| const host = process.env.TAURI_DEV_HOST; | |||
| const __filename = fileURLToPath(import.meta.url); | |||
| const __dirname = dirname(__filename); | |||
| // https://vitejs.dev/config/ | |||
| export default defineConfig(async () => ({ | |||
| plugins: [vue()], | |||
| // Vite options tailored for Tauri development and only applied in `tauri dev` or `tauri build` | |||
| // | |||
| // 1. prevent vite from obscuring rust errors | |||
| clearScreen: false, | |||
| // 2. tauri expects a fixed port, fail if that port is not available | |||
| server: { | |||
| port: 1420, | |||
| strictPort: true, | |||
| host: host || false, | |||
| hmr: host | |||
| ? { | |||
| protocol: "ws", | |||
| host, | |||
| port: 1421, | |||
| } | |||
| : undefined, | |||
| watch: { | |||
| // 3. tell vite to ignore watching `src-tauri` | |||
| ignored: ["**/src-tauri/**"], | |||
| }, | |||
| }, | |||
| resolve: { | |||
| alias: { | |||
| "@": path.resolve(__dirname, "src"), // 使用绝对路径 | |||
| }, | |||
| }, | |||
| })); | |||
+ 0
- 306
项目测试文档.md
View File
| @ -1,306 +0,0 @@ | |||
| # 📋 WaveControl 隔空手势控制系统 - 测试文档 | |||
| ## 一、测试概述 | |||
| ### 1.1 测试目标 | |||
| 确保系统三个核心子模块功能完整、交互稳定、性能可靠，满足多场景下的非接触式人机交互需求，包括主控制平台、手语通平台、游戏控制模块。 | |||
| ### 1.2 测试对象 | |||
| - 主控制系统（控制面板 + 手势库管理 + 语音识别） | |||
| - WaveSign 手语通系统（教学、评分、社区） | |||
| - 虚拟赛车手柄系统（游戏中控制响应） | |||
| ### 1.3 测试类型 | |||
| | 测试类型 | 说明 | | |||
| | ------------ | ------------------------------------------ | | |||
| | 功能测试 | 各模块是否能完成核心功能 | | |||
| | 集成测试 | 各子模块之间的数据流与交互是否正确 | | |||
| | 性能测试 | 系统是否在高帧率下稳定运行、响应及时 | | |||
| | 边界测试 | 识别模糊手势、断网、摄像头断连等异常情况 | | |||
| | 用户体验测试 | 普通用户是否能流畅使用、易上手、有清晰反馈 | | |||
| ## 二、测试环境 | |||
| | 项目 | 配置 | | |||
| | ----------- | --------------------------------- | | |||
| | 操作系统 | Windows 10 / 11、MacOS | | |||
| | Python 版本 | Python 3.8+ | | |||
| | 浏览器 | Chrome / Edge | | |||
| | 识别设备 | USB 外接摄像头 / 笔记本自带摄像头 | | |||
| | 游戏平台 | Steam 平台《Rush Rally Origins》 | | |||
| ## 三、测试用例设计 | |||
| ### ✅ 主控制系统 | |||
| | 序号 | 手势名称 | 手势动作说明 | 所属类型 | | |||
| | ---- | ------------ | --------------------------------------------- | -------- | | |||
| | 01 | 光标控制 | 竖起食指滑动控制光标位置 | 通用控制 | | |||
| | 02 | 鼠标左键点击 | 食指 + 大拇指上举执行点击 | 通用控制 | | |||
| | 03 | 滚动控制 | okay 手势（食指+拇指捏合），上下移动滚动页面 | 通用控制 | | |||
| | 04 | 全屏控制 | 四指并拢向上 → 触发设定键（默认 f 键） | 通用控制 | | |||
| | 05 | 退格 | 特定手势触发退格键 | 通用控制 | | |||
| | 06 | 开始语音识别 | 六指手势触发语音识别启动 | 通用控制 | | |||
| | 07 | 结束语音识别 | 拳头手势触发语音识别停止 | 通用控制 | | |||
| | 08 | 暂停/继续 | 单手张开保持 1.5 秒触发暂停/继续识别 | 通用控制 | | |||
| | 09 | 向右移动 | 拇指上抬，其余手指收回 → 控制游戏角色向右移动 | 游戏控制 | | |||
| | 10 | 跳跃 | 食指、中指上举 → 控制跳跃动作 | 游戏控制 | | |||
| | 11 | 右跳跃 | 拇指 + 食指 + 中指上举 → 控制右跳跃 | 游戏控制 | | |||
| | 12 | 上一首 | 大拇指向左摆动 → 上一首音乐 | 音乐控制 | | |||
| | 13 | 下一首 | 大拇指向右摆动 → 下一首音乐 | 音乐控制 | | |||
| | 14 | 暂停/播放 | 比耶手势（✌️ ） → 暂停或播放音乐 | 音乐控制 | | |||
| | 15 | 切换音乐模式 | rock 手势（🤘）→ 切换音乐/普通控制模式 | 模式切换 | | |||
| ### 🤟 手语通 WaveSign | |||
| #### ✅ 1. **SLClassroom（手语教室）模块** | |||
| | 用例编号 | 用例名称 | 测试点 | 预期结果 | | |||
| | -------- | ------------------ | --------------------------- | --------------------------- | | |||
| | TC-SL-01 | 摄像头实时识别 | 摄像头接通后手势是否被识别 | 返回手语内容 + 实时评分动画 | | |||
| | TC-SL-02 | 视频上传评分 | 上传手语视频后是否正常评分 | 返回分数、标准建议 | | |||
| | TC-SL-03 | 视频课程学习流程 | 是否能顺序播放、标记已学 | 视频播放正常，课程解锁 | | |||
| | TC-SL-04 | 翻转卡片练习 | 卡片是否翻转 + 显示正确答案 | 点击翻面后显示预设解释 | | |||
| | TC-SL-05 | 任务式课程地图跳转 | 点击课程节点是否正确跳转 | 跳转至对应课程页 | | |||
| #### ✅ 2. **Community（社区系统）模块** | |||
| | 用例编号 | 用例名称 | 测试点 | 预期结果 | | |||
| | --------- | ------------- | -------------------------- | -------------------------- | | |||
| | TC-COM-01 | 发帖功能 | 输入文字/图片/视频发帖 | 帖子成功展示 + ID 唯一标识 | | |||
| | TC-COM-02 | 评论功能 | 帖子下评论 + 删除 | 评论正常显示/删除 | | |||
| | TC-COM-03 | 点赞机制 | 点赞后数值变化 | 点赞数+1，重复点则取消 | | |||
| | TC-COM-04 | 热门话题显示 | 帖子互动数高时是否上热门区 | 热门区出现帖子 | | |||
| | TC-COM-05 | 内容审核机制 | 敏感词是否被拦截/提示 | 给出“内容不合规”提示 | | |||
| | TC-COM-06 | 标签推荐 | 选择话题是否推荐相关内容 | 推荐结果合理、及时加载 | | |||
| | TC-COM-07 | 关注/取关系统 | 关注后是否成功建立关注关系 | 动态展示更新 | | |||
| #### ✅ 3. **Schedule（日程与任务模块）** | |||
| | 用例编号 | 用例名称 | 测试点 | 预期结果 | | |||
| | --------- | -------------------- | -------------------------- | ------------------------- | | |||
| | TC-SCH-01 | 添加任务清单 | 是否可设置日期/优先级 | 列表显示任务 + 状态可勾选 | | |||
| | TC-SCH-02 | 事件提醒触发 | 设置提醒是否能按时通知 | 到时弹出提醒/响铃提示 | | |||
| | TC-SCH-03 | 日/周/月视图切换 | 是否能无误切换不同日历视图 | 各视图正常显示 | | |||
| | TC-SCH-04 | 删除任务是否更新视图 | 删除后是否同步更新 | 日历图/列表同步清除 | | |||
| #### ✅ 4. **LifeServing（生活服务）模块** | |||
| | 用例编号 | 用例名称 | 测试点 | 预期结果 | | |||
| | -------- | ---------------- | ---------------------------------- | ---------------------------- | | |||
| | TC-LS-01 | 发布内容管理 | 发布好物推荐/活动等信息 | 内容展示无误 | | |||
| | TC-LS-02 | 辅助器具推荐 | 推荐列表是否分类清晰/加载正确 | 分类显示 + 图片正常 | | |||
| | TC-LS-03 | 学习设备推荐 | 展示硬件学习工具列表 | 内容图文加载完整 | | |||
| | TC-LS-04 | 就业信息推送 | 职位内容、公司信息展示是否完整 | 包含岗位名称、描述、联系方式 | | |||
| | TC-LS-05 | 残障友好企业标识 | 是否加V / 标签区分 | 有“友好企业”提示 | | |||
| | TC-LS-06 | 无障碍路线规划 | 是否能绘制无台阶/电梯路线 | 地图路径合理 | | |||
| | TC-LS-07 | 实时公交提醒 | 路线是否加载正确/更新是否及时 | 实时展示公交进站时间 | | |||
| | TC-LS-08 | 活动预告展示 | 亲子/文娱/演出分类信息加载是否完整 | 可报名 + 有时间地点说明 | | |||
| #### ✅ 5. **MyPage（个人中心）模块** | |||
| | 用例编号 | 用例名称 | 测试点 | 预期结果 | | |||
| | -------- | ------------------ | ------------------------------- | ---------------------------- | | |||
| | TC-MY-01 | 修改头像 | 上传头像 + 预览功能是否生效 | 显示新头像 | | |||
| | TC-MY-02 | 修改资料 | 昵称/简介/联系方式可修改 | 保存后页面同步更新 | | |||
| | TC-MY-03 | 收藏管理 | 收藏帖子/课程后是否能查看 | 我的收藏页显示内容 | | |||
| | TC-MY-04 | 账号注册/登录/登出 | 是否可注册新用户 + 正确跳转状态 | 新用户进入首页，旧账号可退出 | | |||
| | TC-MY-05 | 权限角色识别 | 是否区分普通用户/审核员角色 | 页面显示不同选项 | | |||
| #### ✅ 6. **技术实现相关（稳定性/架构）测试** | |||
| | 用例编号 | 用例名称 | 测试点 | 预期结果 | | |||
| | ---------- | --------------------- | ----------------------------- | ------------------------ | | |||
| | TC-TECH-01 | SQLite 数据读写测试 | 批量操作课程/帖子是否存取正常 | 数据不丢失，响应时间正常 | | |||
| | TC-TECH-02 | MediaPipe模型崩溃恢复 | 强行关闭摄像头后是否重连 | 显示重连提示/自动恢复 | | |||
| | TC-TECH-03 | Tailwind 前端样式响应 | 各分辨率下页面是否响应式变化 | 不溢出，元素自适应 | | |||
| ### 🏎️ 游戏控制模块 | |||
| | 用例编号 | 模块 | 用例名称 | 前置条件 | 测试步骤 | 预期结果 | 优先级 | | |||
| | ---------- | -------- | ---------------- | ------------------------ | -------------------------------- | ---------------------------------- | ------ | | |||
| | TC-GAME-01 | 游戏控制 | 加速动作识别 | 摄像头运行正常，程序启动 | 举起右手拇指上扬 | 车辆开始加速 | 高 | | |||
| | TC-GAME-02 | 游戏控制 | 刹车动作识别 | 同上 | 举起左手拇指上扬 | 车辆开始减速 | 高 | | |||
| | TC-GAME-03 | 游戏控制 | 左右转向动作识别 | 同上 | 向左倾手掌 | 车辆向左转弯 | 高 | | |||
| | TC-GAME-04 | 游戏控制 | 手势连续识别切换 | 程序运行中 | 快速从加速 → 左转 → 刹车切换手势 | 每步操作都有反馈，游戏响应流畅 | 高 | | |||
| | TC-GAME-05 | 游戏控制 | 识别抖动干扰测试 | 程序运行中，抖动手指 | 快速小幅度摆动手指 | 系统不误触操作，保持稳定 | 中 | | |||
| | TC-GAME-06 | 游戏控制 | 虚拟手柄断连恢复 | 手柄模拟开启中 | 断开 vgamepad 端口 → 重连 | 程序检测中断并尝试自动恢复 | 中 | | |||
| | TC-GAME-07 | 游戏控制 | 低帧率下性能表现 | 模拟20fps摄像头 | 尝试完成左右转、加速等动作 | 出现识别滞后，界面给出低帧警告 | 中 | | |||
| | TC-GAME-08 | 游戏控制 | UI反馈准确性 | 摄像头运行中 | 做出加速动作，观察 UI 面板反馈 | 显示“当前动作：加速”，图像区域标亮 | 中 | | |||
| ## 四、测试程序实现与技术支撑 | |||
| 本项目测试除使用主系统 UI 操作外，亦开发了两套独立测试程序用于识别稳定性、输入准确性与边界场景的验证，覆盖核心逻辑路径，支撑高频回归测试与离线分析。 | |||
| ### 4.1 手势识别测试程序（wavecontrol-test 模块） | |||
| 项目测试脚本集中存放于 `wavecontrol-test/src/` （gesture分支）路径下，采用 TypeScript + Vue3 框架实现，通过 MediaPipe 实时检测与手势逻辑模块协同，实现系统功能验证。 | |||
| #### 4.1.1 hand_landmark 模块 | |||
| - **detector.ts** | |||
| 核心手部关键点检测模块，封装对 MediaPipe 的调用逻辑，统一输出手部21个关键点的坐标、置信度等数据。 | |||
| - 功能点：初始化摄像头流、绑定回调、封装模型参数。 | |||
| - 用于：为 `VideoDetector.vue` 和 `gesture_handler.ts` 提供关键点数据源。 | |||
| - **gesture_handler.ts** | |||
| 手势解析与事件派发模块，将 landmark 数据解析为具体手势动作（如光标控制、点击、跳跃等）。 | |||
| - 支持自定义手势库扩展。 | |||
| - 与游戏控制模块或系统控制指令绑定。 | |||
| - **VideoDetector.vue** | |||
| Vue 组件封装，展示摄像头实时画面 + 可视化 landmark 点位（调试模式用）。 | |||
| - 提供测试 UI 面板，便于调试每个手势识别过程。 | |||
| - 集成 FPS 状态、实时识别手势结果反馈。 | |||
| #### 4.1.2 独立运行说明与调试提示 | |||
| - **模块定位：** | |||
| `wavecontrol-test` 为独立测试工程，当前未集成至主项目的 UI 页面路由体系，主要用于**手势识别逻辑的单元测试与调试验证**。 | |||
| - **运行状态说明：** | |||
| 启动后页面加载为空白（白屏），属**正常表现**，原因如下： | |||
| - 项目尚未渲染任何主界面组件； | |||
| - 测试逻辑运行主要依赖控制台输出来验证关键逻辑（如 landmark 检测、手势判断等）。 | |||
| - **调试建议：** | |||
| 启动后打开浏览器开发者工具（快捷键 `F12` 或 `Ctrl+Shift+I`），查看： | |||
| - 控制台（Console）：打印识别结果、错误日志、手势状态； | |||
| - 网络（Network）：确认模型文件是否成功加载； | |||
| - 元素（Elements）：手动挂载 `VideoDetector.vue`注入测试组件进行临时渲染调试。 | |||
| ### 4.2 游戏控制模块测试（`test.py` 等） | |||
| ### 4.2.1 模块定位与结构 | |||
| 该模块为游戏控制核心动作识别的测试环境，主要用于模拟真实场景下用户的手势输入，评估系统能否准确识别特定动作（如加速、转弯、刹车等），并通过 OpenCV 实时可视化手势状态与角度变化。 | |||
| - **所在目录：** `wavecontrol/test_py/`（`game_control` 分支） | |||
| - **相关脚本：** | |||
| - `test.py`：主测试入口，执行实时手势识别与状态展示。 | |||
| - `gesture_detector.py`：封装具体的手势识别逻辑。 | |||
| - `utils.py`：通用工具类，如角度计算、数据格式处理等。 | |||
| - `test_xbox.py`：拟用于连接虚拟 Xbox 控制器，模拟游戏输入。 | |||
| - `test_gui.py`：图形界面测试入口。 | |||
| ### 4.2.2 核心测试逻辑（基于 `test.py`） | |||
| **功能点解析：** | |||
| - **手势状态追踪：** | |||
| 通过 `gesture_status` 字典记录五类状态： | |||
| - `Left Fist`、`Right Fist`：用于映射刹车/加速等动作 | |||
| - `Left Thumb`、`Right Thumb`：映射转向或音量等功能 | |||
| - `Angle`：表示当前手势的偏转角度，用于模拟方向盘行为或特殊动作 | |||
| - **视觉反馈：** | |||
| 使用 `OpenCV` (`cv2`) 实时将识别结果叠加在摄像头图像上，显示五种状态的当前值，方便快速人工验证是否识别准确。 | |||
| - **模块化调用：** | |||
| `GestureDetector` 实例通过 `gesture_detector.py` 导入，使得识别逻辑可独立测试、方便集成到主游戏模块。 | |||
| ------ | |||
| #### 4.2.3 使用说明 | |||
| - **启动方法：** | |||
| ``` | |||
| python test.py | |||
| ``` | |||
| **运行效果：** | |||
| - 打开摄像头窗口； | |||
| - 实时显示识别到的五项手势状态； | |||
| - 开发者可通过手部动作验证识别准确性。 | |||
| - **测试目标：** | |||
| - 验证关键手势在自然使用状态下的识别准确率； | |||
| - 评估系统在不同光照/距离/角度条件下对“加速”“刹车”“左右转”的响应稳定性； | |||
| - 检查 OpenCV 图像反馈是否与真实操作一致，便于回归对比。 | |||
| ## 五、测试结果分析 | |||
| #### 项目整体手动测试结果： | |||
| [⌛️项目测试结果](./项目测试结果.xlsx) | |||
| #### 主平台自动化测试结果： | |||
| 全部测试通过，覆盖率达79% | |||
|  | |||
| #### WaveSign平台自动化测试结果： | |||
| 全部测试通过，覆盖率达93% | |||
| 具体测试设计和结果见以下链接： | |||
| [WaveSign测试文档](https://gitee.com/wydhhh/software-engineering/blob/wavesign/WaveSign%E6%B5%8B%E8%AF%95%E6%96%87%E6%A1%A3.md) | |||
| [WaveSign测试结果html报告](https://gitee.com/wydhhh/software-engineering/tree/wavesign/htmlcov) | |||
| ### **5.1 测试通过率** | |||
| - **总体通过率：** | |||
| 在所有模块（主控制系统 + WaveSign + 游戏控制模块）共计 **50+ 条用例 × 10次测试 = 500+ 次执行记录**中，约 **88%** 的测试结果为“通过”，**12%** 的测试记录显示“未通过”。 | |||
| - **通过率较高的模块：** | |||
| - **主控制系统：**通过率 **>90%**，大部分基础手势（如光标控制、鼠标点击、滚动控制）表现稳定。 | |||
| - **WaveSign 社区与个人中心模块：**通过率 **约95%**，发帖、评论、关注、资料修改等常规操作无明显问题。 | |||
| - **通过率较低的模块：** | |||
| - **WaveSign SLClassroom（手语教室）：**实时识别和视频上传评分测试中偶尔出现 **识别延迟** 或 **评分不稳定**，通过率约 **80%**。 | |||
| - **游戏控制模块：**“低帧率下性能表现”与“识别抖动干扰测试”未通过率稍高（**20% 左右**），主要原因是低帧率摄像头模拟下识别滞后明显。 | |||
| ------ | |||
| ### **5.2 典型问题分析** | |||
| - **（1）识别类问题：** | |||
| - 当背景复杂或光照不足时，MediaPipe 模型识别准确率下降，个别测试记录显示 **响应时间超过0.7s**，甚至未正确识别手势。 | |||
| - “六指手势启动语音识别”在部分测试中未触发，需要优化手势阈值。 | |||
| - **（2）性能问题：** | |||
| - 游戏模块在模拟 **20fps 低帧率**摄像头时，出现识别滞后，UI延迟反馈。 | |||
| - 部分 UI 动画在高分辨率（4K）设备上存在轻微卡顿，需要进一步优化前端渲染。 | |||
| - **（3）社区系统：** | |||
| - 敏感词拦截逻辑偶尔误报，如普通词语被识别为敏感词。 | |||
| - **（4）任务提醒功能：** | |||
| - **日/周/月视图切换**测试中发现，在多任务快速切换时，少数场景下界面刷新不完全。 | |||
| ------ | |||
| ### **5.3 综合结论** | |||
| - WaveControl 系统整体功能 **满足预期目标**，大部分核心功能已稳定实现。 | |||