FAQs
If you encounter any problems during the use of the product, you can refer to this document.
Register and Login Related
Q: What should I do if my registration is not approved?
Users need to scan the QR code through WeChat, fill in the enterprise email, phone number, mobile verification code, and supplement complete personal information to register on the platform. Only after the administrator approves can the platform be used for model and algorithm evaluation. The administrator's approval results will be notified through personal email. If it is not approved, please follow the instructions in the email to supplement information or modify the email to submit for approval again.
Image Related
Q: Does the platform support custom build images?
Currently, the platform only supports user to import built images, and does not support the use of dockerfile to build images on the platform. The dockerfile filled in by users is only used for platform approvers to review images. For specific image import related content, please refer to the user manual.
Q: What should I do if the imported image has been in import process for a long time?
The image import time depends on the size of the image and whether it is speed-limited. Generally, it takes about 30 minutes to 2 hours to import images. If the image is too large to pull, please contact the administrator and give it to the administrator in another method (such as a tar package), and the administrator will import it into the image repository.
Q: Are there any special requirements for images imported into the platform at present?
Currently, the platform supports Ubuntu image and Centos image. If you have other images, please contact the administrator for corresponding processing.
In addition, currently the platform only supports downloading images without login. Before importing images, please try to pull it locally. If the following error is returned, it means that login is required to download. Please migrate the image to a repository that does not require login and try again.
File Upload Related
Q: How to upload models and files?
Users can upload files to be evaluated such as models, code, and data by installing the FlagEval-Serving tool. Different domains require different directory structures and interface specifications for evaluation. Please refer to the requirements of each domain to prepare the documents to be uploaded.
Note: the directory structures and specifications for NLP, CV, Multimodal, Audio, and innovative algorithm verification are different. Please prepare the relevant files strictly according to the given examples, otherwise the evaluation will fail.
Q: What should I do if re-uploading the file fails?
- For the uploaded model files, if you only want to modify some of the non-model files, please upload the files to be modified with the original token. Or directly edit editable files or create new small files on the web page.
- If you need to re-upload all files, please open [Upload Model & Code] t get a new user token, and use
flageval-serving upload --token=TOKEN demoto upload.
Q: How to upload additional files?
Common commands for flageval-serving are as follows:
- Uploading updated files requires a unique token for confirmation.
- Click [Upload Model & Code] on the evaluation details page to get a token for initializing the creation of the evaluation directory.
- If the user appends a file in the current version, use the cp command ad update it with the token of the file list page.
- Note: the following commands are all in English single quotes.
| Command | Meaning | Notes |
|---|---|---|
| pip install --upgrade flageval-serving | Update to the latest version | Please update to the latest version to use |
| flageval-serving upload --token='new token' /demo | Upload the local demo folder to the FlagEval platform, where the 'new token' is obtained in the evaluation details page [Upload Model & Code]. |  |
| flageval-serving cp --token='old token' /temp /remote | Append the local /temp folder to the /remote directory used by the 'old token'. The effect is the same as the cp directory in linux:
| Only the old token of the file details page can be used. Cannot use a new token. |
| flageval-serving ls --token='old token' /remote | List directories and files under platform's /remote | |
| flageval-serving | Output help-related commands |
Evaluation Task Related
Q: How is the number of evaluations calculated?
Each user usually only has 5 formal evaluation opportunities per month (including model evaluation and innovative algorithm verification). For model evaluation, it is recommended to ensure that the code can run successfully through [Inference Verification] before conducting formal evaluation. If there are special evaluation needs that require more opportunities, please contact the administrator to add them.
Please note: if there is a failure in the formal evaluation, clicking [Continue] will not deduct opportunities. Only clicking [Formal Inference Evaluation] will deduct opportunities. If the failure is caused by platform reasons, you can contact the administrator to return the deducted opportunities.
Q: What should I do if there is an error during training due to insufficient shared memory (shm) for innovative algorithm verification?
The value of shared content (/dev/shm) defaults to half of the memory value of the resource request. If it is not enough, you can enable this option to increase its value, but the value must be less than the value of the requested memory. As shown in the figure below, the shared content (/dev/shm) must be less than 80.
Q: Does it always show queuing after launching the inference service?
This may be due to the shortage of platform computing resources. For special tasks that require urgent evaluation, please contact the administrator to arrange computer power.
Q: What is the purpose of algorithm verification?
Algorithm verification is mainly to verify the reproducibility of innovative algorithms in the project. Only 1000 iterations are selected for verification, and only the data required from these 1000 iterations is uploaded.
Leaderboard Related
Q: No response after submitting to the leaderboard?
Currently, users cannot submit their scores to the leaderboard. Please contact the administrator if necessary.