Competition

Formatting the submission file:

The output of your method submitted to the evaluation system should be a single CSV file with the following columns:

1. PatientID: string - patient identifier
2. StudyUID: string - study identifier
3. View: string - view name, one of: RLL, LCC, RMLO, LMLO
4. X: integer - X coordinate (on the horizontal axis) of the left edge of the predicted bounding box in 0-based indexing (for the left-most column of the image x=0)
5. Width: integer - predicted bounding box width (along the horizontal axis)
6. Y: integer - Y coordinate (on the vertical axis) of the top edge of the predicted bounding box in 0-based indexing (for the top-most column of the image y=0)
7. Height: integer - predicted bounding box height (along the vertical axis)
8. Z: integer - the first bounding box slice number in 0-based indexing (for the first slice of the image z=0)
9. Depth: integer - predicted bounding box slice span (size along the depth axis)
10. Score: float - predicted bounding box confidence score on arbitrary scale, unified across all the cases within a single submission (e.g. 0.0 – 1.0)

Ex:

PatientID,StudyUID,View,X,Width,Y,Height,Z,Depth,Score ID1,UID1,RLL,X(int),Width(int),Y(int),Height(int),Z(int),Depth(int),Score(float) ID2,UID2,LCC,X(int),Width(int),Y(int),Height(int),Z(int),Depth(int),Score(float) ID3,UID3,RMLO,X(int),Width(int),Y(int),Height(int),Z(int),Depth(int),Score(float) ID4,UID4,LMLO,X(int),Width(int),Y(int),Height(int),Z(int),Depth(int),Score(float)

Coordinates of the predicted bounding boxes should be given for the correct image orientation. In the official competition GitHub repository, we provide a python function for loading image data from a DICOM file into 3D array of pixel values.

Each entry (row) in the submission file must correspond to exactly one predicted bounding box. There may be arbitrary number of predicted bounding boxes for each DBT volume. It is not required to have predictions for all DBT volumes.

Example submission file will be provided on the competition website.

Definition of a true-positive detection

A predicted box is going to be counted as a true positive if the distance in pixels in the original image between its center point and the center of a ground truth box is less than half of its diagonal or 100 pixels, whichever is larger.

In terms of the third dimension, the ground truth bounding box is assumed to span 25% of volume slices before and after the ground truth center slice and the predicted box center slice is required to be included in this range to be considered a true positive.

Performance metric:

The overall performance will be assessed as the average sensitivity for 1, 2, 3, and 4 FP/s per volume. The competition performance will be assessed only on studies containing a biopsied lesion.

Participants may use the training set cases in any manner they would like for the purpose of training their systems (consistent with the data license); there will be no restrictions on the advice sought from local experts for training purposes. The participants can also combine the provided training data with other data if they disclose that in the description of the algorithm.

The validation set and test sets cases, however, are to be manipulated, processed, and analyzed without human intervention.

Participants are free to download the training set and, subsequently, the validation and test sets when these datasets become available. It is important to note that once participants submit their test set output to the challenge organizers, they will be considered fully vested in the challenge, so that their performance results (without links to the identity of the participant) will become part of any presentations, publications, or subsequent analyses derived from the challenge at the discretion of the organizers.

Submissions of test set output will not be considered complete and valid for final competition standings unless they are accompanied by a high-level description of (1) the model/algorithms used (but no code or technical specifics required) and (2) the data used to create/train the model. Additionally, by submitting test results, you agree that the data of such results, including bounding boxes, metrics and other derivatives thereof, are allowed to be used alongside the aforementioned model and data descriptions for the competition's accompanying publication prepared by competition organizers, with the possibility of such results being made publicly available and associated with your team. You also agree to be acknowledged in this publication by name and institution.

Members of top teams will be invited to prepare and co-author this paper, a joint publication on this competition alongside the competition organizers. This manuscript will describe the details of the challenge itself, as well as the methods used by the participants and the corresponding results. However, please not that this is not a condition to participate.

Challenges are designed to motivate and reward novel computational approaches to a defined task. The use of commercial software (unless your group is affiliated with the organization that holds intellectual property rights to that software) or open source software (unless your group has a recognized association with the creation of that software) is not allowed, unless you can clearly demonstrate an innovative use, alteration, or enhancement to the application of such software.

Participation in the DBTex Challenges acknowledges the educational, friendly competition, and community-building nature of the challenges and commits to conduct consistent with this spirit for the advancement of the medical imaging research community. See this article for a discussion of lessons learned from the LUNGx Challenge, also sponsored by SPIE, AAPM, and NCI.

Terms and Conditions

By participating in this challenge, each participant agrees to:

1. Detect lesions that were subsequently sent to biopsy. Those include both cancers and benign lesions.
2. All participants must attest that they are not directly affiliated with the labs of any of the DBTex2 organizers or major contributors. Please refer to the Challenge Organizer Guidance document of the AAPM Working Group on Grand Challenges here.
3. All participants must submit a high level description of model and data used for their test submissions to accompany said submission. Additionally, by submitting, participants agree that their test results and derived metrics can be made publically available and associated with their team, to be used for publication by the competition organizers in the manuscript that will discuss the competition.
4. The participants are encouraged to use this data beyond this competition and consistently with the use conditions described on the TCIA website https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=64685580). The use of the data should be acknowledged as described on the TCIA website.