# VLM ## 3D Scene Understanding ### Data prepration Download json files from [here](https://huggingface.co/AIDC-AI/Omni-View/tree/main/eval_dataset). Move the json files to `./dataset/eval/3dscene/` Download metadata from [EmbodiedScan](https://github.com/OpenRobotLab/EmbodiedScan/tree/main/data). You need to fill out the official form to get the access to the dataset. Move the `embodiedscan_infos_*.pkl` to `./dataset/eval/embodiedscan`. Download images from [Video3DLLM](https://huggingface.co/datasets/zd11024/Video-3D-LLM_data). Then, ```shell cd Video-3D-LLM_data # unzip posed images cat posed_images_part* > posed_images.tar.gz tar -xzf posed_images.tar.gz # unzip mask unzip mask.zip # unzip pcd tar -xzf pcd_with_object_aabbs.tar.gz mkdir scannet mv posed_images/ scannet/ mv mask/ scannet/ mv data/scannet/pcd_with_object_aabbs/ scannet/ ``` Move the `scannet` to `./dataset/eval/`. The whole file structure under `./dataset/eval/` will be as follows. ```shell ./dataset/eval/ ├── 3dscene │ ├── scannet_det_val_4frames.json │ ├── scannet_select_frames.json │ ├── scanqa_val_llava_style.json │ ├── scanrefer_val_32frames.json │ └── sqa3d_test_llava_style.json ├── embodiedscan │ ├── embodiedscan_infos_test.pkl │ ├── embodiedscan_infos_train.pkl │ └── embodiedscan_infos_val.pkl └── scannet ├── mask ├── pcd_with_object_aabbs └── posed_images 6 directories, 9 files ``` ### Inference ```shell torchrun --nproc_per_node=4 --master_port=12345 -m eval.vlm.eval.vqa.evaluate_3dvqa --model-path ./pretrained_model/BAGEL-7B-MoT/ --safetensor-path model.safetensors --dataset sqa3d torchrun --nproc_per_node=4 --master_port=12345 -m eval.vlm.eval.vqa.evaluate_3dvqa --model-path ./pretrained_model/BAGEL-7B-MoT/ --safetensor-path model.safetensors --dataset scanqa torchrun --nproc_per_node=4 --master_port=12345 -m eval.vlm.eval.vqa.evaluate_3dvqa --model-path ./pretrained_model/BAGEL-7B-MoT/ --safetensor-path model.safetensors --dataset scanrefer torchrun --nproc_per_node=4 --master_port=12345 -m eval.vlm.eval.vqa.evaluate_3dvqa --model-path ./pretrained_model/BAGEL-7B-MoT/ --safetensor-path model.safetensors --dataset 3ddet ``` The results (`*.json` files) will be saved in `./results/`. ### Evaluation #### SQA3D ```shell python eval/vlm/eval/vqa/3dvqa_eval.py --dataset sqa3d --input-file ./results/sqa3d.json # output EM-all: 59.16453537936914 EM-what: 51.787271142109844 EM-which: 50.427350427350426 EM-can: 68.04733727810651 EM-is: 73.15950920245399 EM-how: 60.86021505376345 EM-others: 56.71378091872792 EM-R-all: 62.432509235578294 EM-R-what: 58.326068003487364 EM-R-which: 51.566951566951566 EM-R-can: 68.04733727810651 EM-R-is: 75.61349693251533 EM-R-how: 61.29032258064516 EM-R-others: 59.8939929328622 ``` #### ScanQA ```shell python eval/vlm/eval/vqa/3dvqa_eval.py --dataset scanqa --input-file ./results/scanqa.json # output CIDER: 103.01606173581641 BLEU: 48.355630543562654, 32.50084859567164, 23.329984202585262, 16.194281050888 METEOR: 20.101447665118403 Rouge: 49.04130603336627 EM: 29.497326203208555 ``` #### ScanRefer ```shell python eval/vlm/eval/vqa/3dgrounding_eval.py --input-file ./results/scanrefer.json # output all iou@0.25: 50.82036180058898 all iou@0.5: 45.0462768195204 multiple iou@0.25: 44.877985123319846 multiple iou@0.5: 39.34490408456218 unique iou@0.25: 75.50135501355012 unique iou@0.5: 68.72628726287263 ``` #### 3D Detection ```shell python eval/vlm/eval/vqa/3ddet_eval.py --input-file ./results/3ddet.json # output +Metrics Per Category------+--------+----------+ | Category | Precision | Recall | F1 Score | +--------------+-----------+--------+----------+ | chair | 0.4807 | 0.5074 | 0.4937 | | pillow | 0.1683 | 0.1881 | 0.1777 | | cabinet | 0.1401 | 0.1608 | 0.1497 | | table | 0.3744 | 0.3528 | 0.3633 | | lamp | 0.1302 | 0.0986 | 0.1122 | | couch | 0.4795 | 0.4862 | 0.4828 | | desk | 0.3567 | 0.3896 | 0.3724 | | stand | 0.4167 | 0.3509 | 0.3810 | | bed | 0.7468 | 0.6797 | 0.7117 | | backpack | 0.3130 | 0.3628 | 0.3361 | | bathtub | 0.4343 | 0.4000 | 0.4164 | | ottoman | 0.1250 | 0.0714 | 0.0909 | | dresser | 0.4828 | 0.3825 | 0.4268 | | bin | 0.3727 | 0.3162 | 0.3421 | | toilet | 0.7720 | 0.7395 | 0.7554 | | refrigerator | 0.3486 | 0.4176 | 0.3800 | | stove | 0.7826 | 0.7347 | 0.7579 | | microwave | 0.2453 | 0.1884 | 0.2131 | | monitor | 0.2422 | 0.2770 | 0.2585 | | computer | 0.1546 | 0.0968 | 0.1190 | | window | 0.1297 | 0.0997 | 0.1127 | | shelf | 0.1939 | 0.2184 | 0.2054 | | curtain | 0.1260 | 0.1291 | 0.1275 | | plant | 0.1538 | 0.0855 | 0.1099 | | stairs | 0.3243 | 0.4000 | 0.3582 | | picture | 0.0212 | 0.0212 | 0.0212 | | book | 0.0348 | 0.0629 | 0.0448 | | bottle | 0.0247 | 0.0284 | 0.0264 | | lamp | 0.1302 | 0.0986 | 0.1122 | | towl | 0.0000 | 0.0000 | 0.0000 | | sink | 0.4752 | 0.4467 | 0.4605 | +--------------+-----------+--------+----------+ +--------+---------------+------------+--------+ | Split | Avg Precision | Avg Recall | Avg F1 | +--------+---------------+------------+--------+ | cate8 | 0.4751 | 0.4553 | 0.4644 | | cate20 | 0.3783 | 0.3601 | 0.3670 | | cate31 | 0.2961 | 0.2836 | 0.2877 | +--------+---------------+------------+--------+ ``` > During evaluation, the error `ERROR | __main__:threedod_process_results:357 - Error parsing prediction bbox` may appear. This error does not affect the evaluation. ## VSI-Bench ### Data prepration Download [VSI-Bench](https://huggingface.co/datasets/nyu-visionx/VSI-Bench), put it in `dataset/eval`. ### Inference ```shell torchrun --nproc_per_node=4 --master_port=12345 -m eval.vlm.eval.vqa.evaluate_vsibench --model-path ./pretrained_model/BAGEL-7B-MoT/ --safetensor-path model.safetensors --dataset vsibench ``` The results (`*.json` files) will be saved in `./results/`. ### Evaluation ```shell python eval/vlm/eval/vqa/3dvqa_eval.py --dataset vsibench --input-file ./results/vsibench.json # output obj_appearance_order_accuracy: 49.029126213592235 object_abs_distance_MRA:.5:.95:.05: 46.402877697841724 object_counting_MRA:.5:.95:.05: 70.31858407079646 object_rel_distance_accuracy: 65.91549295774648 object_size_estimation_MRA:.5:.95:.05: 68.59391395592864 room_size_estimation_MRA:.5:.95:.05: 54.722222222222214 route_planning_accuracy: 33.50515463917525 object_rel_direction_accuracy: 54.404572390100945 overall: 55.3614930184255 ``` ## Novel View Synthesis ### Data prepration Download the RealEstate10K dataset from [this link](http://schadenfreude.csail.mit.edu:8000/), which is provided by [pixelSplat](https://github.com/dcharatan/pixelsplat), and `unzip` the zip file and put the data in `YOUR_RAW_DATAPATH`. Run the following command to preprocess the data into our format. ```shell git clone https://github.com/zalkklop/LVSM.git cd LVSM python process_data.py --base_path YOUR_RAW_DATAPATH --output_dir ./dataset/eval/re10k/ --mode ['train' or 'test'] ``` The whole file structure under `./dataset/eval/re10k/test/` will be as follows. ``` ./dataset/eval/re10k/test/ ├── full_list.txt ├── images │ ├── 000c3ab189999a83 │ ├── ... ├── metadata │ ├── 000c3ab189999a83.json │ ├── ... ``` ### Evaluation We provide a script to evaluate Omni-View on [RE10k](https://google.github.io/realestate10k/). ```shell python inference.py --scene-id 000c3ab189999a83 ``` | Argument | Default | Description | | ---------------------------- | ---------------------- | ---------------------------------------------------------------- | | `scene-id` | None | The scene id in RE10k. | | `pose-id` | None | The id of camera trajectory in RE10k. Default: pose_id = scene_id | | `image-path` | None | The reference image path. | If `scene-id != pose-id`, we will use the first image of scene-id as the reference image and generate novel views using the camera trajectory of pose-id. If `(scene-id is None) and (image-path is not None)`, we will use the image in image-path as the reference image and generate novel views using the camera trajectory of pose-id.