MoE-Adapters4CL
논문 "Mixture-of-Experts 어댑터를 통한 비전-언어 모델의 연속 학습 향상" CVPR2024의 코드입니다.목차
초록
Continual learning can empower vision-language models to continuously acquire new knowledge, without the need for access to the entire historical dataset. However, mitigating the performance degradation in large-scale models is non-trivial due to (i) parameter shifts throughout lifelong learning and (ii) significant computational burdens associated with full-model tuning. In this work, we present a parameter-efficient continual learning framework to alleviate long-term forgetting in incremental learning with vision-language models. Our approach involves the dynamic expansion of a pre-trained CLIP model, through the integration of Mixture-of-Experts (MoE) adapters in response to new tasks. To preserve the zero-shot recognition capability of vision-language models, we further introduce a Distribution Discriminative Auto-Selector (DDAS) that automatically routes in-distribution and out-of-distribution inputs to the MoE Adapter and the original CLIP, respectively. Through extensive experiments across various settings, our proposed method consistently outperforms previous state-of-the-art approaches while concurrently reducing parameter training burdens by 60%.접근법
___
설치
conda create -n MoE_Adapters4CL python=3.9
conda activate MoE_Adapters4CL
conda install pytorch==2.1.0 torchvision==0.16.0 torchaudio==2.1.0 pytorch-cuda=11.8 -c pytorch -c nvidia
cd cil
pip install -r requirements.txt데이터 준비
대상 데이터셋: Aircraft, Caltech101, CIFAR10, CIFAR100, DTD, EuroSAT, Flowers, Food, MNIST, OxfordPet, StanfordCars, SUN397, TinyImagenet.Caltech101에 문제가 있을 경우 issue#6를 참고할 수 있습니다.
자세한 내용은 ZSCL의 datasets.md를 참조하세요. 멋진 작업을 해주신 분들께 큰 감사를 드립니다!
모델 체크포인트
| | 모델 | 링크 | |------------------|----------------------------------------------------------------------|---------------------------------------------------------------------- | | full_shot_order1 | full_shot_order1_1000iters.pth | Baidu Disk / Google Drive | | few_shot_order1 | few_shot_order1_1000iters.pth | Baidu Disk / Google Drive |MTCL
테스트 단계
예시:- 체크포인트를 MoE-Adapters4CL/ckpt 폴더로 이동
- ``
cd MoE-Adapters4CL/mtil3. 스크립트 실행`bash srcipts/test/Full_Shot_order1.sh`
학습 단계
예시:- 체크포인트를 MoE-Adapters4CL/ckpt 폴더로 이동
- `
cd MoE-Adapters4CL/mtil` - 스크립트 실행 `
bash srcipts/train/train_full_shot_router11_experts22_1000iters.sh`
클래스 증분 학습
학습 단계
예시:- `
cd cil` - `
bash run_cifar100-2-2.sh`
인용문헌
@inproceedings{yu2024boosting, title={Boosting continual learning of vision-language models via mixture-of-experts adapters}, author={Yu, Jiazuo and Zhuge, Yunzhi and Zhang, Lu and Hu, Ping and Wang, Dong and Lu, Huchuan and He, You}, booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition}, pages={23219--23230}, year={2024} } ``
감사의 글
저희 저장소는 wise-ft, Continual-CLIP 및 ZSCL을 기반으로 구축되었습니다. 코드를 공유해 주신 저자분들께 감사드립니다.--- Tranlated By Open Ai Tx | Last indexed: 2025-12-04 ---