Upload benchmarking data to S3 with Neuroglancer

[1]:
from brainlit.utils import upload
from pathlib import Path
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-1-04530185d659> in <module>
----> 1 from brainlit.utils import upload
      2 from pathlib import Path

ModuleNotFoundError: No module named 'brainlit'

Uploading Benchmarking Images from local data locations .

This notebook demonstrates uploading the benchmarking data and associated .swc segment files. The upload destination could easily be set to a url of a cloud data server such as s3.

1) Define variables.

  • source is the root directory of the data and swc files.

    • the .tif file is in the root directory and .swc files are in a subfolder called “consensus-swcs”

  • p is the prefix string. file:// indicates a filepath, while s3:// or gc:// indicate URLs.

  • dest and dest_segments are the destinations for the uploads (in this case, filepaths).

The below paths lead to sample data in my local drive. Alter the below path definitions to point to your own local file locations.

Note:

The below upload destination points to the open-neurodata S3. Uploading data will overwrite the current benchmarking data on S3.

[2]:
source = (Path().resolve().parents[5] / "Downloads" / "validation_21").as_posix()
dest = "s3://open-neurodata/brainlit/benchmarking_data/validation_21"
dest_segments = "s3://open-neurodata/brainlit/benchmarking_data/validation_21"
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-2-8447993c36e2> in <module>
----> 1 source = (Path().resolve().parents[5] / "Downloads" / "validation_21").as_posix()
      2 dest = "s3://open-neurodata/brainlit/benchmarking_data/validation_21"
      3 dest_segments = "s3://open-neurodata/brainlit/benchmarking_data/validation_21"

NameError: name 'Path' is not defined

2) Upload the segmentation data (.swc)

[3]:
upload.upload_segments(source, dest_segments, 1, benchmarking = True)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-3-716b8fe203a0> in <module>
----> 1 upload.upload_segments(source, dest_segments, 1, benchmarking = True)

NameError: name 'upload' is not defined

3) Upload the image data (.tif)

[4]:
upload.upload_volumes(source, dest, 1, benchmarking = True)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-4-552f14221402> in <module>
----> 1 upload.upload_volumes(source, dest, 1, benchmarking = True)

NameError: name 'upload' is not defined

Appendix

  • If when downloading, you get a reshape error, try uploading the segments before uploading the volumes

[5]:
test_list = []
validation_list = []
for i in range(25):
    test_list.append('test_' + str(i+1))
    validation_list.append('validation_' + str(i+1))
[6]:
num_res = 1

for test in test_list:
    print(test)
    source = (Path().resolve().parents[5] / "Downloads" / test).as_posix()
    dest = "s3://open-neurodata/brainlit/benchmarking_data/" + test
    dest_segments = "s3://open-neurodata/brainlit/benchmarking_data/" + test
    upload.upload_segments(source, dest_segments, num_res, benchmarking = True)
    upload.upload_volumes(source, dest, num_res, benchmarking = True)

for val in validation_list:
    print(val)
    source = (Path().resolve().parents[5] / "Downloads" / val).as_posix()
    dest = "s3://open-neurodata/brainlit/benchmarking_data/" + val
    dest_segments = "s3://open-neurodata/brainlit/benchmarking_data/" + val
    upload.upload_segments(source, dest_segments, num_res, benchmarking = True)
    upload.upload_volumes(source, dest, num_res, benchmarking = True)
test_1
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-6-503aad61967a> in <module>
      3 for test in test_list:
      4     print(test)
----> 5     source = (Path().resolve().parents[5] / "Downloads" / test).as_posix()
      6     dest = "s3://open-neurodata/brainlit/benchmarking_data/" + test
      7     dest_segments = "s3://open-neurodata/brainlit/benchmarking_data/" + test

NameError: name 'Path' is not defined