Frequently Asked Questions (FAQ) for the 2024 Challenge

This page provides specific FAQs for the 2024 Challenge. Please read the general Challenge FAQs for more general questions about the Challenges.

Rules

Data

Scoring

Computation

Submissions

Rules

I missed the abstract deadline. Can I still participate in the Challenge?

Yes, you can still participate. An accepted CinC abstract is required for prizes, rankings, and the opportunity to present your work at CinC, but you can still submit algorithms to the official phase without an accepted abstract.

A ‘wildcard’ entry is reserved for a high-scoring team who submitted an abstract that was not accepted or who were unable to submit an abstract by the deadline. Please read here for more details, including the deadline.

Can I make the license open source but restrict commercial use?

Yes, the philosophy of the Challenge is to encourage researchers to make their code free to use for research. We hope that companies will approach you to license the code, too! If you do not specify any license, then we will assume that the license is the BSD 3-Clause License.

Do we need to provide our training code?

Yes, this is a required (and exciting) part of this year’s Challenge.

Can I provide my training code but request that you not use/run it?

No, the training code is an important part of this year’s Challenge.

Am I allowed to do transfer learning using pre-trained networks?

Yes, most certainly. We encourage you to do this. You do not need to include your data in the code stack for training the algorithm, but you do need to include the pre-trained model in the code and provide code to retrain (continue training) on the training data we provide. You must also thoroughly document the content of the database you used to pre-train. You cannot use transfer learning to avoid training your model.

Data

Are the training data representative of the validation and test data?

Yes, to a degree, but the validation and test sets do not have all of the information included in the training data, specifically the labels!

Why are the training, validation, and test data drawn from slightly different populations?

This is deliberate - it reflects the real world. Your algorithms will be used on humans in the future, and they will look different to humans now. One major challenge is to create an algorithm that generalizes to new data. We check this by evaluating the algorithms on held-out data with somewhat different populations. You can also do this by sourcing more of your own data, or building in physiological and clinical knowledge.

Do I need to upload your training data? What about the code for evaluating my algorithm?

No, we have the training, validation, and test data as well as the evaluation code.

Scoring

What did you choose the Challenge metric? I think it is flawed or could be improved.

The Challenge organizers are always looking for feedback from the teams/public, particularly during the first phase of the Challenge from January to April. After that period, we update the Challenge rules to address the public commentary. The metrics are never perfect, but we do try to create meaningful metrics that are relevant to healthcare, which is why we rarely use the usual information retrieval metrics. We often attempt to build in the clinical response, the medical resources available, and the relative (health and financial) costs of false vs. true positives and negatives. In general, the top 5-10 teams are often comparable. We believe the discussion that we generate around the problem, the heterogeneity of approaches, and the optimization of domain-aware metrics is more important than any single winner.

Are the scores currently on the leaderboard the final scores for the Challenge?

No, the leaderboard contains scores on the validation data during the unofficial and official phases of the Challenge. The final scores on the test data will be released after the conference for the preferred model selected by each team.

How do I choose which my submission for evaluation on the test data?

You will be able to choose which model you would like to have scored on the test set. We will ask for teams to choose their preferred model shortly after the end of the official phase of the Challenge. If you do not choose a model, or if there is any ambiguity about your choice, then we will use the model with the highest score on the validation data.

Computation

What computational resources do you provide for our code?

We are using a g4dn.4xlarge instance on AWS or a comparable environment to run your code. It has 16 vCPUs, 64 GB RAM (60 GB available to your code), 300 GB of local storage (in addition to the data), and an optional NVIDIA T4 GPU.

For training your model on the training set, we impose a 24 hour time limit on a subset (~1000 records) of the training set and a 168 hour limit on the entirety (~22,000 records) of the training set. For running your trained model on the validation set (~1000 records), we impose a 24 hour time limit. For running your trained model on the test set, we impose a similar limit as the validation set, but with more time for more records, e.g., 48 hours if the test set is twice the size of the validation set.

Will our entry have network access? What if we need to download something?

No, the execution environment has no network access. However, there is network access while we are building your Docker image. If you need to download something, you can put a download command in the Dockerfile.

Can I save something to the container’s filesystem during training and access it during inference?

No, the filesystem is not persistent between stages. You can write to the container’s filesystem temporarily, but those changes will be lost when we finish running the container. If you want to save something during training, it can go in the model folder that gets passed as an argument to the train_model script.

If my submission times out, can you restore the state of the container and continue from there?

No, if we restart it we will have to run your code from the beginning.

Is there write access to the data folder?

No, data is always read-only. If you want to make changes to the data, you will have to make a copy and edit the copy.

What are the requirements for the Docker container?

Please make sure the Docker container meets these requirements:

Submissions

Should I submit your example code to test the submission system?

No, please only submit your code to the submission system.

Should I submit an empty repository to test the submission system?

No, please only submit an entry after you have finished and tested your code.

I left out a file, or I missed the deadline, or something else. Can I email you my code?

No, please use the submission form to submit your entry through a repository.

Can I change the folder name for the pretrained model?

No, please leave it as model. Our system will always look there for a model.

Do you run the code that was in my repository at the time of submission?

Yes, we pull the latest version of the code that was in the repository before the time of submission.

Why is my entry unsuccessful on your submission system? It works on my computer.

If you used Python for your entry, then please test it in Docker. See the submissions page for details.

My entry had some kind of error. Did I lose one of my total entries?

No, only scored entries (submitted entries that receive a score) count against the total number of allowed entries.

For more general Challenge FAQs, please visit here.

Supported by the National Institute of Biomedical Imaging and Bioengineering (NIBIB) under NIH grant number R01EB030362.

© PhysioNet Challenges. Website content licensed under the Creative Commons Attribution 4.0 International Public License.

Back