Compare commits
merge into: 10235501447:master
10235501447:finalv1
10235501447:finalv2
10235501447:finalv3
10235501447:game_control
10235501447:gesture
10235501447:gesture-game
10235501447:gesture_for_chrome
10235501447:master
10235501447:screenshot
10235501447:vosk_inc
10235501447:wavesign
10235501447:web
pull from: 10235501447:gesture
10235501447:finalv1
10235501447:finalv2
10235501447:finalv3
10235501447:game_control
10235501447:gesture
10235501447:gesture-game
10235501447:gesture_for_chrome
10235501447:master
10235501447:screenshot
10235501447:vosk_inc
10235501447:wavesign
10235501447:web
| Author | SHA1 | Message | Date |
|---|---|---|---|
 Backpack
|
8bfb8be26f | update readme3.0 | 2 months ago |
|
Backpack
|
dd23b169de | update readme | 2 months ago |
|
Backpack
|
15434c3f2e | finish scroll | 2 months ago |
|
Backpack
|
bda10ef218 | send enter fix | 2 months ago |
|
Backpack
|
d1691ae0cb | send enter | 2 months ago |
|
Backpack
|
27a882d29e | mouse control and click | 2 months ago |
|
Backpack
|
cd75372433 | other gestures | 2 months ago |
|
Backpack
|
f74228e552 | THREE_FINGERS_UP | 2 months ago |
25 changed files with 1895 additions and 115 deletions
Split View
Diff Options
-
+9 -0.gitignore
-
+108 -16README.md
-
+5 -4wavecontrol-test/index.html
-
+754 -1wavecontrol-test/package-lock.json
-
+7 -1wavecontrol-test/package.json
-
+43 -0wavecontrol-test/src-py/main.py
-
+0 -0wavecontrol-test/src-py/router/__init__.py
-
BINwavecontrol-test/src-py/router/__pycache__/__init__.cpython-312.pyc
-
BINwavecontrol-test/src-py/router/__pycache__/__init__.cpython-38.pyc
-
BINwavecontrol-test/src-py/router/__pycache__/ws.cpython-312.pyc
-
BINwavecontrol-test/src-py/router/__pycache__/ws.cpython-38.pyc
-
+225 -0wavecontrol-test/src-py/router/ws.py
-
+41 -24wavecontrol-test/src/hand_landmark/VideoDetector.vue
-
+104 -22wavecontrol-test/src/hand_landmark/detector.ts
-
+341 -5wavecontrol-test/src/hand_landmark/gesture_handler.ts
-
+49 -0wavecontrol-test/src/locales/en.ts
-
+18 -0wavecontrol-test/src/locales/i18n.ts
-
+47 -0wavecontrol-test/src/locales/zh.ts
-
+10 -4wavecontrol-test/src/main.ts
-
+87 -0wavecontrol-test/src/store/app.ts
-
+7 -0wavecontrol-test/src/vite-env.d copy.ts
-
+9 -0wavecontrol-test/test_mouse.py
-
+0 -15wavecontrol-test/tsconfig.app.json
-
+28 -5wavecontrol-test/tsconfig.json
-
+3 -18wavecontrol-test/tsconfig.node.json
+ 9
- 0
.gitignore
View File
| @ -0,0 +1,9 @@ | |||
| .vscode/ | |||
| node_modules/ | |||
| src-py/__pycache__/ | |||
| dist/ | |||
| build/ | |||
| model/ | |||
| .vite/ | |||
| src-tauri/ | |||
| db.sqlite3 | |||
+ 108
- 16
README.md
View File
| @ -1,37 +1,129 @@ | |||
| # 隔空手势识别系统 | |||
| # WaveControl 手势识别测试模块 | |||
| ### 项目简介 | |||
| > 本目录为主项目的**手势检测与识别子系统测试环境**，用于独立调试、验证基于 MediaPipe 的手部 landmark 检测与自定义手势逻辑的正确性。 | |||
| 本项目是一套基于**摄像头手势识别 + 语音识别**的非接触式人机交互系统，旨在帮助用户**在无需触碰设备的情况下完成对电脑的基本操作**。 | |||
| ------ | |||
| 灵感源于日常生活中“手脏无法点击”、“操作不便”的场景，结合后疫情时代对无接触操作的广泛需求，系统通过**手势识别模块**感知用户动作，通过**语音识别模块**解析用户指令，并联动前端和后端的控制逻辑，完成**系统级的键盘/鼠标模拟操作**，打造自然、高效的交互体验。 | |||
| ## 模块结构 | |||
| ``` | |||
| wavecontrol-test/ | |||
| ├── public/ # 静态资源 | |||
| ├── src/ # 前端源代码（Vue3 + TS） | |||
| │ ├── components/ # UI组件（如 VideoDetector） | |||
| │ ├── hand_landmark/ # 核心手势识别模块（detector.ts、gesture_handler.ts） | |||
| │ └── locales/ # 多语言支持（如有） | |||
| ├── src-py/ # Python 辅助测试脚本（如 test_mouse.py） | |||
| ├── index.html # 主入口页面 | |||
| ├── main.ts # 启动脚本 | |||
| ├── vite.config.ts # 前端构建配置 | |||
| ├── tsconfig*.json # TypeScript 配置 | |||
| └── README.md # 本文件 | |||
| ``` | |||
| ------ | |||
| ### 项目周期 | |||
| ## 功能说明 | |||
| 本项目为 2025 年《软件工程实践》课程的结课实践项目，采用**敏捷开发流程**，共分为四次迭代，逐步完成从原型设计到系统实现，最终交付一个可运行的完整系统，实现稳定、实用的**非接触式交互体验**。 | |||
| 本模块聚焦**手部关键点检测 + 自定义手势识别逻辑的单元验证**，具备如下能力： | |||
| | 功能点 | 描述 | | |||
| | --------------- | ------------------------------------------------------------ | | |||
| | 摄像头接入 | 调用浏览器摄像头，获取实时图像流 | | |||
| | MediaPipe 集成 | 通过 `detector.ts` 接入 Google 手部识别模型，输出 21 个关键点数据 | | |||
| | 手势判断 | 通过 `gesture_handler.ts` 识别特定手势，如拇指上扬、握拳、转动角度等 | | |||
| | 可视化展示 | `VideoDetector.vue` 渲染摄像头画面 + landmark 点位，辅助调试 | | |||
| | 控制台输出 | 打印每帧检测结果（landmark 坐标、识别状态、FPS） | | |||
| | Python 模拟测试 | `test_mouse.py` 用于鼠标控制测试，独立于前端进行快速逻辑验证 | | |||
| ### 项目成员 | |||
| 王云岱 朱子玥 杨嘉莉 | |||
| ## 实现手势功能一览 | |||
| 模块共识别并支持 **15种核心手势动作**，覆盖通用控制、游戏互动、音乐控制等多种场景，现阶段已完成如下识别逻辑与事件映射： | |||
| ### 主控制系统 | |||
| ### 核心功能 | |||
| | 序号 | 手势名称 | 动作说明 | 所属类型 | | |||
| | ---- | ------------ | ---------------------------------------------- | -------- | | |||
| | 01 | 光标控制 | 竖起食指滑动控制光标位置 | 通用控制 | | |||
| | 02 | 鼠标左键点击 | 食指 + 拇指上举触发点击事件 | 通用控制 | | |||
| | 03 | 滚动控制 | okay 手势（食指+拇指捏合）上下移动控制页面滚动 | 通用控制 | | |||
| | 04 | 全屏控制 | 四指并拢向上 → 触发设定键（默认 F 键） | 通用控制 | | |||
| | 05 | 退格 | 特定手势触发退格键 | 通用控制 | | |||
| | 06 | 开始语音识别 | 六指（含小指）手势启动语音识别 | 通用控制 | | |||
| | 07 | 结束语音识别 | 握拳触发语音识别终止 | 通用控制 | | |||
| | 08 | 暂停/继续 | 单手张开静止 1.5 秒，触发系统暂停或继续识别 | 通用控制 | | |||
| ### 技术架构 | |||
| ### 游戏控制 | |||
| | 序号 | 手势名称 | 动作说明 | 所属类型 | | |||
| | ---- | -------- | ------------------------------------------- | -------- | | |||
| | 09 | 向右移动 | 拇指上抬，其余手指收回 → 控制游戏角色向右走 | 游戏控制 | | |||
| | 10 | 跳跃 | 食指 + 中指上举，触发跳跃操作 | 游戏控制 | | |||
| | 11 | 右跳跃 | 拇指 + 食指 + 中指上举，触发右跳跃组合动作 | 游戏控制 | | |||
| ### 应用场景 | |||
| - 📺 追剧、吃饭时无需碰鼠标 | |||
| - 👨🏫 教学演讲中通过手势控制幻灯片 | |||
| - 👨🍳 厨房中翻菜谱、视频食谱不沾油手 | |||
| - 🏥 医疗等无菌场景下控制系统界面 | |||
| - 🤖 AR/智慧眼镜等未来设备交互前置尝试 | |||
| ### 音乐控制 | |||
| | 序号 | 手势名称 | 动作说明 | 所属类型 | | |||
| | ---- | --------- | ------------------------------------ | -------- | | |||
| | 12 | 上一首 | 拇指向左摆动，控制播放器切换至上一首 | 音乐控制 | | |||
| | 13 | 下一首 | 拇指向右摆动，切换到下一首音乐 | 音乐控制 | | |||
| | 14 | 暂停/播放 | 比耶手势（✌️ / 🤘）控制音乐播放或暂停 | 音乐控制 | | |||
| ### 模式切换 | |||
| | 序号 | 手势名称 | 动作说明 | 所属类型 | | |||
| | ---- | ------------ | ---------------------------------------------- | -------- | | |||
| | 15 | 切换音乐模式 | rock 手势（🤘）触发音乐控制模式与普通模式的切换 | 模式切换 | | |||
| 📌 **说明**： | |||
| - 所有手势动作识别基于 `MediaPipe` 提供的 21 关键点； | |||
| - 每个手势映射事件已封装在 `gesture_handler.ts` 中； | |||
| - 可通过修改 `gesture_map` 动态扩展或调整触发逻辑； | |||
| - 模块支持平滑滤波、多帧确认机制，显著减少误识别。 | |||
| ## 快速启动（开发模式） | |||
| ``` | |||
| cd wavecontrol-test | |||
| npm install | |||
| npm run dev | |||
| ``` | |||
| > 默认运行在 http://localhost:5173，**页面为空白属正常现象**，请使用浏览器开发者工具调试。 | |||
| ------ | |||
| ## 调试指南 | |||
| - 打开开发者工具（F12）： | |||
| - **Console**：查看关键日志（手势判断结果、识别帧率等） | |||
| - **Network**：确认资源加载是否成功（如 MediaPipe 模型） | |||
| - **Elements**：手动挂载 `VideoDetector` 测试组件进行可视化验证 | |||
| > 此模块**未接入主 UI 路由体系**，可作为“独立沙箱”运行，便于开发期间快速验证识别效果。 | |||
| ------ | |||
| ## 典型用例 | |||
| - ✅ 验证 `gesture_handler.ts` 中的规则逻辑是否准确 | |||
| - ✅ 查看 landmark 点是否能稳定输出（避免帧丢失） | |||
| - ✅ 测试新手势添加逻辑的正确性 | |||
| - ✅ 结合 Python 脚本快速验证控制指令映射逻辑 | |||
+ 5
- 4
wavecontrol-test/index.html
View File
| @ -1,13 +1,14 @@ | |||
| <!doctype html> | |||
| <html lang="en"> | |||
| <html lang="en" style="height: 100%;width: 100%;"> | |||
| <head> | |||
| <meta charset="UTF-8" /> | |||
| <link rel="icon" type="image/svg+xml" href="/vite.svg" /> | |||
| <meta name="viewport" content="width=device-width, initial-scale=1.0" /> | |||
| <title>Vite + Vue + TS</title> | |||
| <title>Tauri + Vue + Typescript App</title> | |||
| </head> | |||
| <body> | |||
| <div id="app"></div> | |||
| <body style="height: 100%;width: 100%;overflow: hidden;"> | |||
| <div id="app" style="height: 100%;width: 100%"></div> | |||
| <script type="module" src="/src/main.ts"></script> | |||
| </body> | |||
| </html> | |||
+ 754
- 1
wavecontrol-test/package-lock.json
View File
+ 7
- 1
wavecontrol-test/package.json
View File
+ 43
- 0
wavecontrol-test/src-py/main.py
View File
| @ -0,0 +1,43 @@ | |||
| import os | |||
| import sys | |||
| import uvicorn | |||
| from fastapi import FastAPI | |||
| from fastapi.middleware.cors import CORSMiddleware | |||
| from router.ws import router as ws_router | |||
| if hasattr(sys, 'frozen'): | |||
| os.chdir(os.path.dirname(sys.argv[0])) | |||
| app = FastAPI() | |||
| app.add_middleware( | |||
| CORSMiddleware, | |||
| allow_origins=["*"], | |||
| allow_credentials=True, | |||
| allow_methods=["*"], | |||
| allow_headers=["*"], | |||
| ) | |||
| # 添加 WebSocket 路由 | |||
| app.include_router(ws_router) | |||
| @app.get("/") | |||
| def read_root(): | |||
| return "ready" | |||
| @app.get("/shutdown") | |||
| def shutdown(): | |||
| import signal | |||
| import os | |||
| os.kill(os.getpid(), signal.SIGINT) | |||
| if __name__ == '__main__': | |||
| port = 62334 | |||
| print(f"Starting server at http://127.0.0.1:{port}/docs") | |||
| uvicorn.run(app, host="127.0.0.1", port=port) | |||
+ 0
- 0
wavecontrol-test/src-py/router/__init__.py
View File
BIN
wavecontrol-test/src-py/router/__pycache__/__init__.cpython-312.pyc
View File
BIN
wavecontrol-test/src-py/router/__pycache__/__init__.cpython-38.pyc
View File
BIN
wavecontrol-test/src-py/router/__pycache__/ws.cpython-312.pyc
View File
BIN
wavecontrol-test/src-py/router/__pycache__/ws.cpython-38.pyc
View File
+ 225
- 0
wavecontrol-test/src-py/router/ws.py
View File
| @ -0,0 +1,225 @@ | |||
| import json | |||
| from dataclasses import dataclass | |||
| from typing import Dict, List, Optional, Union | |||
| from fastapi import APIRouter, WebSocket, WebSocketDisconnect | |||
| from pynput.keyboard import Controller as KeyboardController | |||
| from pynput.keyboard import Key | |||
| from pynput.mouse import Button, Controller | |||
| router = APIRouter() | |||
| # 存储活跃的WebSocket连接 | |||
| active_connection: Optional[WebSocket] = None | |||
| @dataclass | |||
| class WebSocketMessage: | |||
| """WebSocket消息数据类""" | |||
| type: str | |||
| msg: str = "" | |||
| title: str = "提示" | |||
| duration: int = 1 | |||
| data: Dict = None | |||
| def __post_init__(self): | |||
| if self.data is None: | |||
| self.data = {"x": 0, "y": 0, "key_str": ""} | |||
| def to_dict(self) -> Dict: | |||
| """将消息转换为字典格式""" | |||
| return { | |||
| "type": self.type, | |||
| "msg": self.msg, | |||
| "title": self.title, | |||
| "duration": self.duration, | |||
| "data": self.data, | |||
| } | |||
| class WebSocketMessageType: | |||
| """WebSocket消息类型常量""" | |||
| # 系统消息类型 | |||
| INFO = "info" | |||
| SUCCESS = "success" | |||
| WARNING = "warning" | |||
| ERROR = "error" | |||
| # 鼠标操作类型 | |||
| MOUSE_MOVE = "mouse_move" | |||
| MOUSE_CLICK = "mouse_click" | |||
| MOUSE_SCROLL_UP = "mouse_scroll_up" | |||
| MOUSE_SCROLL_DOWN = "mouse_scroll_down" | |||
| # 键盘操作类型 | |||
| SEND_KEYS = "send_keys" | |||
| class MessageSender: | |||
| """消息发送器,发送给前端""" | |||
| @staticmethod | |||
| async def send_message( | |||
| ws_data_type: str, msg: str, title: str = "提示", duration: int = 1 | |||
| ) -> None: | |||
| """ | |||
| 发送消息到WebSocket客户端 | |||
| Args: | |||
| ws_data_type: 消息类型 | |||
| msg: 消息内容 | |||
| title: 消息标题 | |||
| duration: 显示持续时间 | |||
| """ | |||
| if not active_connection: | |||
| return | |||
| try: | |||
| message = WebSocketMessage( | |||
| type=ws_data_type, msg=msg, title=title, duration=duration | |||
| ) | |||
| await active_connection.send_json(message.to_dict()) | |||
| except Exception as e: | |||
| print(f"发送消息失败: {e}") | |||
| class GestureHandler: | |||
| """手势控制器""" | |||
| def __init__(self): | |||
| self.keyboard = KeyboardController() | |||
| self.mouse = Controller() | |||
| def move_mouse(self, x: int, y: int) -> None: | |||
| """移动鼠标到指定位置""" | |||
| print(f"👉 正在移动鼠标到位置: ({x}, {y})") | |||
| self.mouse.position = (x, y) | |||
| def click_mouse(self) -> None: | |||
| """点击鼠标左键""" | |||
| print("👉 正在点击鼠标") | |||
| self.mouse.click(Button.left) | |||
| def scroll_up(self) -> None: | |||
| """向上滚动""" | |||
| self.mouse.scroll(0, 1) | |||
| def scroll_down(self) -> None: | |||
| """向下滚动""" | |||
| self.mouse.scroll(0, -1) | |||
| def send_keys(self, key_str: str) -> None: | |||
| """ | |||
| 发送按键事件（支持组合键） | |||
| Args: | |||
| key_str: 按键字符串（如 'ctrl+r' 或 'F11'） | |||
| """ | |||
| try: | |||
| keys = [self.__parse_key(key) for key in key_str.split("+")] | |||
| print("👉 正在点击发送按键") | |||
| self.__execute_keys(keys) | |||
| except Exception as e: | |||
| print(f"发送按键失败: {e}") | |||
| def __parse_key(self, key_str: str) -> Union[str, Key]: | |||
| """ | |||
| 解析单个按键 | |||
| Args: | |||
| key_str: 按键字符串 | |||
| Returns: | |||
| 解析后的按键对象 | |||
| """ | |||
| key_str = key_str.strip().lower() | |||
| if hasattr(Key, key_str): | |||
| return getattr(Key, key_str) | |||
| elif len(key_str) == 1: | |||
| return key_str | |||
| elif key_str.startswith("f"): | |||
| try: | |||
| return getattr(Key, key_str) | |||
| except AttributeError: | |||
| raise ValueError(f"无效的功能键: {key_str}") | |||
| else: | |||
| raise ValueError(f"无效的按键: {key_str}") | |||
| def __execute_keys(self, keys: List[Union[str, Key]]) -> None: | |||
| """ | |||
| 执行按键序列 | |||
| Args: | |||
| keys: 按键列表 | |||
| """ | |||
| pressed_keys = [] | |||
| try: | |||
| # 按下所有键 | |||
| for key in keys: | |||
| self.keyboard.press(key) | |||
| pressed_keys.append(key) | |||
| # 释放所有键（按相反顺序） | |||
| for key in reversed(pressed_keys): | |||
| self.keyboard.release(key) | |||
| except Exception as e: | |||
| print(f"执行按键失败: {e}") | |||
| @router.websocket("/ws_wavecontrol") | |||
| async def websocket_endpoint(websocket: WebSocket): | |||
| """WebSocket端点处理函数""" | |||
| global active_connection | |||
| await websocket.accept() | |||
| active_connection = websocket | |||
| gesture_handler = GestureHandler() | |||
| try: | |||
| while True: | |||
| data_str = await websocket.receive_text() | |||
| await _handle_message(data_str, websocket, gesture_handler) | |||
| except WebSocketDisconnect: | |||
| active_connection = None | |||
| except Exception as e: | |||
| print(f"WebSocket处理错误: {e}") | |||
| active_connection = None | |||
| async def _handle_message( | |||
| data_str: str, | |||
| websocket: WebSocket, | |||
| gesture_handler: GestureHandler, | |||
| ) -> None: | |||
| """处理WebSocket消息""" | |||
| try: | |||
| message = WebSocketMessage(**json.loads(data_str)) | |||
| data = message.data | |||
| print(f"[收到指令] type={message.type}, data={data}") # 调试输出 | |||
| # 处理鼠标操作 | |||
| if message.type == WebSocketMessageType.MOUSE_MOVE: | |||
| gesture_handler.move_mouse(data["x"], data["y"]) | |||
| elif message.type == WebSocketMessageType.MOUSE_CLICK: | |||
| gesture_handler.click_mouse() | |||
| elif message.type == WebSocketMessageType.MOUSE_SCROLL_UP: | |||
| gesture_handler.scroll_up() | |||
| elif message.type == WebSocketMessageType.MOUSE_SCROLL_DOWN: | |||
| gesture_handler.scroll_down() | |||
| # 处理键盘操作 | |||
| elif message.type == WebSocketMessageType.SEND_KEYS: | |||
| gesture_handler.send_keys(data["key_str"]) | |||
| except json.JSONDecodeError: | |||
| print("无效的JSON数据") | |||
| except Exception as e: | |||
| print(f"处理消息失败: {e}") | |||
+ 41
- 24
wavecontrol-test/src/hand_landmark/VideoDetector.vue
View File
| @ -1,24 +1,41 @@ | |||
| import { onMounted, ref } from 'vue'; | |||
| import { Detector } from './hand_landmark/detector'; | |||
| const videoRef = ref<HTMLVideoElement | null>(null); | |||
| const detector = new Detector(); | |||
| onMounted(async () => { | |||
| await detector.initialize(); // 初始化模型 | |||
| const video = videoRef.value!; | |||
| video.width = 640; | |||
| video.height = 480; | |||
| navigator.mediaDevices.getUserMedia({ video: true }).then((stream) => { | |||
| video.srcObject = stream; | |||
| video.play(); | |||
| // 每隔 200ms 检测一帧 | |||
| setInterval(async () => { | |||
| const result = await detector.detect(video); | |||
| await detector.process(result); | |||
| }, 200); | |||
| }); | |||
| }); | |||
| // 不依赖 UI，只需摄像头画面 + 控制逻辑 | |||
| import { createApp, ref, onMounted } from "vue"; | |||
| import { Detector } from "@/hand_landmark/detector"; | |||
| import { GestureHandler } from "@/hand_landmark/gesture_handler"; // 手势处理模块 | |||
| const App = { | |||
| setup() { | |||
| const videoRef = ref<HTMLVideoElement | null>(null); | |||
| const detector = new Detector(); | |||
| const handler = new GestureHandler(); | |||
| onMounted(async () => { | |||
| await detector.initialize(); | |||
| const video = videoRef.value; | |||
| if (!video) return; | |||
| // 获取摄像头 | |||
| const stream = await navigator.mediaDevices.getUserMedia({ video: true }); | |||
| video.srcObject = stream; | |||
| await video.play(); | |||
| // 循环识别 | |||
| const loop = async () => { | |||
| const result = await detector.detect(video); | |||
| const hand = result.rightHand ?? result.leftHand; | |||
| if (hand) { | |||
| const gesture = Detector.getSingleHandGesture(hand); | |||
| handler.handleGesture(gesture, hand); | |||
| } | |||
| requestAnimationFrame(loop); | |||
| }; | |||
| loop(); | |||
| }); | |||
| return () => <video ref={videoRef} width="640" height="480" autoplay />; | |||
| }, | |||
| }; | |||
| createApp(App).mount("#app"); | |||
+ 104
- 22
wavecontrol-test/src/hand_landmark/detector.ts
View File
+ 341
- 5
wavecontrol-test/src/hand_landmark/gesture_handler.ts
View File
+ 49
- 0
wavecontrol-test/src/locales/en.ts
View File
| @ -0,0 +1,49 @@ | |||
| export default { | |||
| "手势识别控制": "Gesture Recognition Control", | |||
| "运行中": "Running", | |||
| "已停止": "Stopped", | |||
| "开机自启动": "Auto Start", | |||
| "显示识别窗口": "Show Recognition Window", | |||
| "摄像头选择": "Camera Selection", | |||
| "手势操作指南": "Gesture Guide", | |||
| "光标控制": "Cursor Control", | |||
| "竖起食指滑动控制光标位置": "Slide with index finger to control cursor position", | |||
| "单击操作": "Click Operation", | |||
| "双指举起执行鼠标单击": "Raise two fingers to perform mouse click", | |||
| "Rock手势执行鼠标单击": "Rock gesture to perform mouse click", | |||
| "滚动控制": "Scroll Control", | |||
| "三指上下滑动控制页面滚动": "Slide three fingers up/down to control page scrolling", | |||
| "（okay手势）食指和拇指捏合滚动页面": "（okay gesture）Pinch with index and thumb to scroll page", | |||
| "食指和拇指距离小于": "Index and thumb distance less than", | |||
| "触发捏合": "Trigger pinch", | |||
| "默认值0.02": "Default value 0.02", | |||
| "可以通过右键->检查->控制台->捏合手势->查看当前距离": "Can check current distance by right-click -> inspect -> console -> pinch gesture", | |||
| "全屏控制": "Full Screen Control", | |||
| "四指并拢发送按键": "Four fingers together to send key", | |||
| "点击设置快捷键": "Click to set shortcut", | |||
| "请按下按键...": "Please press keys...", | |||
| "点击设置": "Click to set", | |||
| "退格": "Backspace", | |||
| "发送退格键": "Send backspace key", | |||
| "开始语音识别": "Start Voice Recognition", | |||
| "六指手势开始语音识别": "Six fingers gesture to start voice recognition", | |||
| "结束语音识别": "End Voice Recognition", | |||
| "拳头手势结束语音识别": "Fist gesture to end voice recognition", | |||
| "暂停/继续": "Pause/Resume", | |||
| "单手张开1.5秒 暂停/继续 手势识别": "Open one hand for 1.5 seconds to pause/resume gesture recognition", | |||
| "识别框x": "Recognition box x", | |||
| "识别框y": "Recognition box y", | |||
| "识别框宽": "Recognition box width", | |||
| "识别框高": "Recognition box height", | |||
| // 通知 | |||
| "WaveControl": "WaveControl", | |||
| "提示": "Tip", | |||
| "停止语音识别": "Stop Voice Recognition", | |||
| "手势识别": "Gesture Recognition", | |||
| "继续手势识别": "Continue Gesture Recognition", | |||
| "暂停手势识别": "Pause Gesture Recognition", | |||
| }; | |||
+ 18
- 0
wavecontrol-test/src/locales/i18n.ts
View File
| @ -0,0 +1,18 @@ | |||
| import { createI18n } from "vue-i18n"; | |||
| // 导入语言包 | |||
| import en from "./en"; | |||
| import zh from "./zh"; | |||
| // 创建 i18n 实例 | |||
| const i18n = createI18n({ | |||
| locale: navigator.language.split("-")[0], // 使用系统语言作为默认语言 | |||
| // locale: "en", // 使用系统语言作为默认语言 | |||
| fallbackLocale: "zh", // 回退语言 | |||
| messages: { | |||
| en, | |||
| zh, | |||
| }, | |||
| }); | |||
| export default i18n; | |||
+ 47
- 0
wavecontrol-test/src/locales/zh.ts
View File
| @ -0,0 +1,47 @@ | |||
| export default { | |||
| "手势识别控制": "手势识别控制", | |||
| "运行中": "运行中", | |||
| "已停止": "已停止", | |||
| "显示识别窗口": "显示识别窗口", | |||
| "摄像头选择": "摄像头选择", | |||
| "手势操作指南": "手势操作指南", | |||
| "开机自启动": "开机自启动", | |||
| "光标控制": "光标控制", | |||
| "竖起食指滑动控制光标位置": "竖起食指滑动控制光标位置", | |||
| "单击操作": "单击操作", | |||
| "双指举起执行鼠标单击": "双指举起执行鼠标单击", | |||
| "Rock手势执行鼠标单击": "Rock手势执行鼠标单击", | |||
| "滚动控制": "滚动控制", | |||
| "三指上下滑动控制页面滚动": "三指上下滑动控制页面滚动", | |||
| "（okay手势）食指和拇指捏合滚动页面": "（okay手势）食指和拇指捏合滚动页面", | |||
| "食指和拇指距离小于": "食指和拇指距离小于", | |||
| "触发捏合": "触发捏合", | |||
| "全屏控制": "全屏控制", | |||
| "四指并拢发送按键": "四指并拢发送按键", | |||
| "点击设置快捷键": "点击设置快捷键", | |||
| "请按下按键...": "请按下按键...", | |||
| "点击设置": "点击设置", | |||
| "退格": "退格", | |||
| "发送退格键": "发送退格键", | |||
| "开始语音识别": "开始语音识别", | |||
| "六指手势开始语音识别": "六指手势开始语音识别", | |||
| "结束语音识别": "结束语音识别", | |||
| "拳头手势结束语音识别": "拳头手势结束语音识别", | |||
| "暂停/继续": "暂停/继续", | |||
| "单手张开1.5秒 暂停/继续 手势识别": "单手张开1.5秒 暂停/继续 手势识别", | |||
| "识别框x": "识别框x", | |||
| "识别框y": "识别框y", | |||
| "识别框宽": "识别框宽", | |||
| "识别框高": "识别框高", | |||
| "默认值0.02": "默认值0.02", | |||
| "可以通过右键->检查->控制台->捏合手势->查看当前距离": "可以通过右键->检查->控制台->捏合手势->查看当前距离", | |||
| // 通知 | |||
| "WaveControl": "WaveControl", | |||
| "提示": "提示", | |||
| "停止语音识别": "停止语音识别", | |||
| "手势识别": "手势识别", | |||
| "继续手势识别": "继续手势识别", | |||
| "暂停手势识别": "暂停手势识别", | |||
| }; | |||
+ 10
- 4
wavecontrol-test/src/main.ts
View File
| @ -1,5 +1,11 @@ | |||
| import { createApp } from 'vue' | |||
| import './style.css' | |||
| import App from './App.vue' | |||
| import { createApp } from "vue"; | |||
| import App from "@/App.vue"; | |||
| createApp(App).mount('#app') | |||
| import { createPinia } from "pinia"; | |||
| const app = createApp(App); | |||
| const pinia = createPinia(); | |||
| app.use(pinia); | |||
| app.mount("#app"); | |||
+ 87
- 0
wavecontrol-test/src/store/app.ts
View File
| @ -0,0 +1,87 @@ | |||
| import { defineStore } from "pinia"; | |||
| interface Camera { | |||
| deviceId: string; | |||
| label: string; | |||
| kind: string; | |||
| } | |||
| enum NotiType { | |||
| INFO = "info", | |||
| SUCCESS = "success", | |||
| WARNING = "warning", | |||
| ERROR = "error", | |||
| } | |||
| export const use_app_store = defineStore("app-store", { | |||
| state: () => ({ | |||
| config: { | |||
| auto_start: false, | |||
| show_window: false, | |||
| four_fingers_up_send: "enter", | |||
| selected_camera_id: "", | |||
| // 识别框 | |||
| boundary_left: 100, | |||
| boundary_top: 100, | |||
| boundary_width: 100, | |||
| boundary_height: 100, | |||
| // 手势识别 | |||
| scroll_gesture_2_thumb_and_index_threshold: 0.02, // 食指和拇指距离阈值 | |||
| }, | |||
| sub_windows: { | |||
| x: 0, | |||
| y: 0, | |||
| progress: 0, | |||
| notification: "", | |||
| noti_type: NotiType.INFO, | |||
| }, | |||
| mission_running: false, | |||
| cameras: [] as Camera[], | |||
| VIDEO_WIDTH: 640, | |||
| VIDEO_HEIGHT: 480, | |||
| flag_detecting: false, | |||
| }), | |||
| // PiniaSharedState 来共享不同 tauri 窗口之间的状态 | |||
| share: { | |||
| // Override global config for this store. | |||
| enable: true, | |||
| initialize: true, | |||
| }, | |||
| actions: { | |||
| is_macos() { | |||
| return navigator.userAgent.includes("Mac"); | |||
| }, | |||
| is_windows() { | |||
| return navigator.userAgent.includes("Windows"); | |||
| }, | |||
| is_linux() { | |||
| return navigator.userAgent.includes("Linux"); | |||
| }, | |||
| async sub_window_info(body: string) { | |||
| this.sub_windows.notification = body; | |||
| this.sub_windows.noti_type = NotiType.INFO; | |||
| }, | |||
| async sub_window_success(body: string) { | |||
| this.sub_windows.notification = body; | |||
| this.sub_windows.noti_type = NotiType.SUCCESS; | |||
| }, | |||
| async sub_window_warning(body: string) { | |||
| this.sub_windows.notification = body; | |||
| this.sub_windows.noti_type = NotiType.WARNING; | |||
| }, | |||
| async sub_window_error(body: string) { | |||
| this.sub_windows.notification = body; | |||
| this.sub_windows.noti_type = NotiType.ERROR; | |||
| }, | |||
| }, | |||
| }); | |||
| export default use_app_store; | |||
+ 7
- 0
wavecontrol-test/src/vite-env.d copy.ts
View File
| @ -0,0 +1,7 @@ | |||
| /// <reference types="vite/client" /> | |||
| declare module "*.vue" { | |||
| import type { DefineComponent } from "vue"; | |||
| const component: DefineComponent<{}, {}, any>; | |||
| export default component; | |||
| } | |||
+ 9
- 0
wavecontrol-test/test_mouse.py
View File
| @ -0,0 +1,9 @@ | |||
| from pynput.mouse import Controller | |||
| import time | |||
| mouse = Controller() | |||
| print("3秒后移动鼠标到位置 (300, 300)...") | |||
| time.sleep(3) | |||
| mouse.position = (300, 300) | |||
| print("鼠标已移动 ✅") | |||
+ 0
- 15
wavecontrol-test/tsconfig.app.json
View File
| @ -1,15 +0,0 @@ | |||
| { | |||
| "extends": "@vue/tsconfig/tsconfig.dom.json", | |||
| "compilerOptions": { | |||
| "tsBuildInfoFile": "./node_modules/.tmp/tsconfig.app.tsbuildinfo", | |||
| /* Linting */ | |||
| "strict": true, | |||
| "noUnusedLocals": true, | |||
| "noUnusedParameters": true, | |||
| "erasableSyntaxOnly": true, | |||
| "noFallthroughCasesInSwitch": true, | |||
| "noUncheckedSideEffectImports": true | |||
| }, | |||
| "include": ["src/**/*.ts", "src/**/*.tsx", "src/**/*.vue"] | |||
| } | |||
+ 28
- 5
wavecontrol-test/tsconfig.json
View File
| @ -1,7 +1,30 @@ | |||
| { | |||
| "files": [], | |||
| "references": [ | |||
| { "path": "./tsconfig.app.json" }, | |||
| { "path": "./tsconfig.node.json" } | |||
| ] | |||
| "compilerOptions": { | |||
| "target": "ES2020", | |||
| "useDefineForClassFields": true, | |||
| "module": "ESNext", | |||
| "lib": ["ES2020", "DOM", "DOM.Iterable"], | |||
| "skipLibCheck": true, | |||
| /* Bundler mode */ | |||
| "moduleResolution": "bundler", | |||
| "allowImportingTsExtensions": true, | |||
| "resolveJsonModule": true, | |||
| "isolatedModules": true, | |||
| "noEmit": true, | |||
| "jsx": "preserve", | |||
| /* Linting */ | |||
| "strict": true, | |||
| "noUnusedLocals": true, | |||
| "noUnusedParameters": true, | |||
| "noFallthroughCasesInSwitch": true, | |||
| "baseUrl": ".", | |||
| "paths": { | |||
| "@/*": ["src/*"] // 将 @ 映射到 src 目录 | |||
| } | |||
| }, | |||
| "include": ["src/**/*.ts", "src/**/*.d.ts", "src/**/*.tsx", "src/**/*.vue"], | |||
| "references": [{ "path": "./tsconfig.node.json" }] | |||
| } | |||
+ 3
- 18
wavecontrol-test/tsconfig.node.json
View File
| @ -1,25 +1,10 @@ | |||
| { | |||
| "compilerOptions": { | |||
| "tsBuildInfoFile": "./node_modules/.tmp/tsconfig.node.tsbuildinfo", | |||
| "target": "ES2023", | |||
| "lib": ["ES2023"], | |||
| "module": "ESNext", | |||
| "composite": true, | |||
| "skipLibCheck": true, | |||
| /* Bundler mode */ | |||
| "module": "ESNext", | |||
| "moduleResolution": "bundler", | |||
| "allowImportingTsExtensions": true, | |||
| "verbatimModuleSyntax": true, | |||
| "moduleDetection": "force", | |||
| "noEmit": true, | |||
| /* Linting */ | |||
| "strict": true, | |||
| "noUnusedLocals": true, | |||
| "noUnusedParameters": true, | |||
| "erasableSyntaxOnly": true, | |||
| "noFallthroughCasesInSwitch": true, | |||
| "noUncheckedSideEffectImports": true | |||
| "allowSyntheticDefaultImports": true | |||
| }, | |||
| "include": ["vite.config.ts"] | |||
| } | |||