Tutorial 4 - Downloading and preparing samples from Recount3#
This tutorial shows how to download real RNA-seq samples from the Recount3 <https://rna.recount.bio/>__ platform and prepare them for use with the bulkdgd model. As a concrete example, we download one TCGA cancer type - glioblastoma multiforme (GBM) - together with three independent SRA studies profiling glioblastoma patient samples (found by searching the SRA metadata for “glioblastoma”/”glioma” and picking studies that represent genuine patient-tumor cohorts). The same steps apply to
any single GTEx tissue, TCGA cancer type, or SRA study - just change the recount3_project_name/recount3_samples_category pairs below.
Once downloaded and preprocessed, the resulting samples can be fed directly into :doc:Tutorial 1 <tutorial_1> to find their representations in the bulkdgd model’s latent space.
[1]:
# Import from the standard library.
import logging as log
# Import from third-party libraries.
import pandas as pd
# Import from 'bulkdgd'.
from bulkdgd import ioutil, recount3
# Set the logging options so that every message of level INFO or above
# is emitted.
log.basicConfig(level = "INFO")
Define the studies#
List the Recount3 studies to download, and where to cache the raw files retrieved from Recount3 (these are not needed once the read counts have been computed).
[2]:
# Define the Recount3 studies we want to download: one TCGA cancer
# type (glioblastoma multiforme, GBM) and three SRA studies profiling
# glioblastoma patient samples (identified by searching the SRA
# metadata for "glioblastoma"/"glioma" and picking studies that survive
# curation as genuine patient-tumor cohorts).
studies = \
[{"recount3_project_name" : "TCGA",
"recount3_samples_category" : "GBM"},
{"recount3_project_name" : "SRA",
"recount3_samples_category" : "SRP027383"},
{"recount3_project_name" : "SRA",
"recount3_samples_category" : "SRP072494"},
{"recount3_project_name" : "SRA",
"recount3_samples_category" : "SRP141440"}]
# Set the directory where the raw files downloaded from Recount3 will
# be cached (they are not needed once the read counts have been
# computed and combined into a single data frame).
raw_wd = "recount3_raw"
Download and convert the studies#
For each study, download its metadata, raw gene counts, and QC metrics from Recount3, then convert the raw counts into read counts by dividing them by each sample’s average mapped read length.
[3]:
# Initialize an empty list to store, for each study, a data frame
# combining the samples' metadata and their gene expression data (as
# read counts).
dfs_studies = []
# For each study
for study in studies:
# Get the name of the project and the samples' category.
project_name = study["recount3_project_name"]
samples_category = study["recount3_samples_category"]
# Get the samples' metadata.
df_metadata = \
recount3.get_metadata(\
project_name = project_name,
samples_category = samples_category,
save_metadata = True,
wd = raw_wd)
# Get the raw gene counts (as base-pair coverage sums).
df_gene_sums = \
recount3.get_gene_sums(\
project_name = project_name,
samples_category = samples_category,
save_gene_sums = True,
wd = raw_wd)
# Get the quality control (QC) metrics, needed to convert the raw
# counts into read counts.
df_qc = \
recount3.get_qc(\
project_name = project_name,
samples_category = samples_category,
save_qc = True,
wd = raw_wd)
# Convert the raw counts into read counts by dividing them by
# each sample's average mapped read length.
df_read_counts = \
recount3.get_read_counts(\
df_raw_counts = df_gene_sums,
avg_mapped_read_length = \
df_qc["star.average_mapped_length"],
do_round = True)
# Combine the metadata with the read counts for this study.
df_study = df_metadata.join(df_read_counts, how = "inner")
# Add the combined data frame to the list.
dfs_studies.append(df_study)
INFO:bulkdgd.recount3.util:Retrieving metadata from: http://duffel.rail.bio/recount3/human/data_sources/tcga/metadata/BM/GBM/tcga.tcga.GBM.MD.gz.
INFO:bulkdgd.recount3.util:The metadata with the sample/experiment attributes split into different columns were successfully written in 'recount3_raw/tcga_GBM_metadata.gz'.
INFO:bulkdgd.recount3.util:The RNA-seq data were successfully saved in 'recount3_raw/tcga_GBM_gene_sums.gz'.
INFO:bulkdgd.recount3.util:The QC metadata were successfully saved in 'recount3_raw/tcga_GBM_qc.gz'.
INFO:bulkdgd.recount3.util:Retrieving metadata from: http://duffel.rail.bio/recount3/human/data_sources/sra/metadata/83/SRP027383/sra.sra.SRP027383.MD.gz.
INFO:bulkdgd.recount3.util:Samples' attributes were found in the metadata (see below).
INFO:bulkdgd.recount3.util:sample attribute 'history' found. Unique values: 'oligodendroastrocytomas', 'oligodendrogliomas', 'recurrent astrocytomas', 'astrocytomas', 'secondary Glioblastomas', 'primary Glioblastomas', 'anaplastic oligodendrogliomas', 'recurrent Glioblastomas', 'recurrent anaplastic oligodendrogliomas', 'anaplastic oligodendroastrocytomas', 'recurrent anaplastic astrocytomas', 'anaplastic astrocytomas', 'recurrent anaplastic oligodendroastrocytomas', 'recurrent oligodendroastrocytomas'.
INFO:bulkdgd.recount3.util:sample attribute 'source_name' found. Unique values: 'Tumor cells'.
INFO:bulkdgd.recount3.util:Experiments' attributes were found in the metadata (see below).
INFO:bulkdgd.recount3.util:experiment attribute 'GEO_Accession' found. Unique values: 'GSM1185864', 'GSM1185865', 'GSM1185866', 'GSM1185867', 'GSM1185868', 'GSM1185869', 'GSM1185870', 'GSM1185871', 'GSM1185872', 'GSM1185873', 'GSM1185874', 'GSM1185875', 'GSM1185876', 'GSM1185877', 'GSM1185878', 'GSM1185879', 'GSM1185880', 'GSM1185881', 'GSM1185882', 'GSM1185883', 'GSM1185884', 'GSM1185885', 'GSM1185886', 'GSM1185887', 'GSM1185888', 'GSM1185889', 'GSM1185890', 'GSM1185891', 'GSM1185892', 'GSM1185893', 'GSM1185894', 'GSM1185895', 'GSM1185896', 'GSM1185897', 'GSM1185898', 'GSM1185899', 'GSM1185900', 'GSM1185901', 'GSM1185902', 'GSM1185903', 'GSM1185904', 'GSM1185905', 'GSM1185906', 'GSM1185907', 'GSM1185908', 'GSM1185909', 'GSM1185910', 'GSM1185911', 'GSM1185912', 'GSM1185913', 'GSM1185914', 'GSM1185915', 'GSM1185916', 'GSM1185917', 'GSM1185918', 'GSM1185919', 'GSM1185920', 'GSM1185921', 'GSM1185922', 'GSM1185923', 'GSM1185924', 'GSM1185925', 'GSM1185926', 'GSM1185927', 'GSM1185928', 'GSM1185929', 'GSM1185930', 'GSM1185931', 'GSM1185932', 'GSM1185933', 'GSM1185934', 'GSM1185935', 'GSM1185936', 'GSM1185937', 'GSM1185938', 'GSM1185939', 'GSM1185940', 'GSM1185941', 'GSM1185942', 'GSM1185943', 'GSM1185944', 'GSM1185945', 'GSM1185946', 'GSM1185947', 'GSM1185948', 'GSM1185949', 'GSM1185950', 'GSM1185951', 'GSM1185952', 'GSM1185953', 'GSM1185954', 'GSM1185955', 'GSM1185956', 'GSM1185957', 'GSM1185958', 'GSM1185959', 'GSM1185960', 'GSM1185961', 'GSM1185962', 'GSM1185963', 'GSM1185964', 'GSM1185965', 'GSM1185966', 'GSM1185967', 'GSM1185968', 'GSM1185969', 'GSM1185970', 'GSM1185971', 'GSM1185972', 'GSM1185973', 'GSM1185974', 'GSM1185975', 'GSM1185976', 'GSM1185977', 'GSM1185978', 'GSM1185979', 'GSM1185980', 'GSM1185981', 'GSM1185982', 'GSM1185983', 'GSM1185984', 'GSM1185985', 'GSM1185986', 'GSM1185987', 'GSM1185988', 'GSM1185989', 'GSM1185990', 'GSM1185991', 'GSM1185992', 'GSM1185993', 'GSM1185994', 'GSM1185995', 'GSM1185996', 'GSM1185997', 'GSM1185998', 'GSM1185999', 'GSM1186000', 'GSM1186001', 'GSM1186002', 'GSM1186003', 'GSM1186004', 'GSM1186005', 'GSM1186006', 'GSM1186007', 'GSM1186008', 'GSM1186009', 'GSM1186010', 'GSM1186011', 'GSM1186012', 'GSM1186013', 'GSM1186014', 'GSM1186015', 'GSM1186016', 'GSM1186017', 'GSM1186018', 'GSM1186019', 'GSM1186020', 'GSM1186021', 'GSM1186022', 'GSM1186023', 'GSM1186024', 'GSM1186025', 'GSM1186026', 'GSM1186027', 'GSM1186028', 'GSM1186029', 'GSM1186030', 'GSM1186031', 'GSM1186032', 'GSM1186033', 'GSM1186034', 'GSM1186035', 'GSM1186036', 'GSM1186037', 'GSM1186038', 'GSM1186039', 'GSM1186040', 'GSM1186041', 'GSM1186042', 'GSM1186043', 'GSM1186044', 'GSM1186045', 'GSM1186046', 'GSM1186047', 'GSM1186048', 'GSM1186049', 'GSM1186050', 'GSM1186051', 'GSM1186052', 'GSM1186053', 'GSM1186054', 'GSM1186055', 'GSM1186056', 'GSM1186057', 'GSM1186058', 'GSM1186059', 'GSM1186060', 'GSM1186061', 'GSM1186062', 'GSM1186063', 'GSM1186064', 'GSM1186065', 'GSM1186066', 'GSM1186067', 'GSM1186068', 'GSM1186069', 'GSM1186070', 'GSM1186071', 'GSM1186072', 'GSM1186073', 'GSM1186074', 'GSM1186075', 'GSM1186076', 'GSM1186077', 'GSM1186078', 'GSM1186079', 'GSM1186080', 'GSM1186081', 'GSM1186082', 'GSM1186083', 'GSM1186084', 'GSM1186085', 'GSM1186086', 'GSM1186087', 'GSM1186088', 'GSM1186089', 'GSM1186090', 'GSM1186091', 'GSM1186092', 'GSM1186093', 'GSM1186094', 'GSM1186095', 'GSM1186096', 'GSM1186097', 'GSM1186098', 'GSM1186099', 'GSM1186100', 'GSM1186101', 'GSM1186102', 'GSM1186103', 'GSM1186104', 'GSM1186105', 'GSM1186106', 'GSM1186107', 'GSM1186108', 'GSM1186109', 'GSM1186110', 'GSM1186111', 'GSM1186112', 'GSM1186113', 'GSM1186114', 'GSM1186115', 'GSM1186116', 'GSM1186117', 'GSM1186118', 'GSM1186119', 'GSM1186120', 'GSM1186121', 'GSM1186122', 'GSM1186123', 'GSM1186124', 'GSM1186125', 'GSM1186126', 'GSM1186127', 'GSM1186128', 'GSM1186129', 'GSM1186130', 'GSM1186131', 'GSM1186132', 'GSM1186133', 'GSM1186134', 'GSM1186135', 'GSM1186136', 'GSM1186137'.
INFO:bulkdgd.recount3.util:The metadata with the sample/experiment attributes split into different columns were successfully written in 'recount3_raw/sra_SRP027383_metadata.gz'.
INFO:bulkdgd.recount3.util:The RNA-seq data were successfully saved in 'recount3_raw/sra_SRP027383_gene_sums.gz'.
INFO:bulkdgd.recount3.util:The QC metadata were successfully saved in 'recount3_raw/sra_SRP027383_qc.gz'.
INFO:bulkdgd.recount3.util:Retrieving metadata from: http://duffel.rail.bio/recount3/human/data_sources/sra/metadata/94/SRP072494/sra.sra.SRP072494.MD.gz.
INFO:bulkdgd.recount3.util:Samples' attributes were found in the metadata (see below).
INFO:bulkdgd.recount3.util:sample attribute 'batch' found. Unique values: 'batch1', 'batch2', 'batch3', 'batch4', 'batch5'.
INFO:bulkdgd.recount3.util:sample attribute 'diagnosis' found. Unique values: 'GBM', 'Secondary_GBM'.
INFO:bulkdgd.recount3.util:sample attribute 'first-line_therapy' found. Unique values: 'RT/TMZ+TMZ', 'RT + TMZ'.
INFO:bulkdgd.recount3.util:sample attribute 'gender' found. Unique values: 'female', 'male'.
INFO:bulkdgd.recount3.util:sample attribute 'library_preperation_date' found. Unique values: '7/10/2014', '8/18/2014', '8/25/2014', '5/8/2014', '4/3/2014'.
INFO:bulkdgd.recount3.util:sample attribute 'rna-extraction_date' found. Unique values: '2/25/2014', '2/21/2014', '8/18/2014'.
INFO:bulkdgd.recount3.util:sample attribute 'sample_from_primary_gbm_diagnosis' found. Unique values: 'no', 'yes'.
INFO:bulkdgd.recount3.util:sample attribute 'source_name' found. Unique values: 'non-responding_before treatment', 'non-responding_after treatment', 'responding_before treatment', 'responding_after treatment'.
INFO:bulkdgd.recount3.util:sample attribute 'subject_group' found. Unique values: 'non-responding', 'responding'.
INFO:bulkdgd.recount3.util:sample attribute 'subject_id' found. Unique values: '102b', '119b', '1305', '1306', '1314', '1326', '22a', '3002', '3003', '3004', '3005', '3010', '3019', '3020', '3021', '32', '35', '2000', '5a', '63b'.
INFO:bulkdgd.recount3.util:sample attribute 'subject_status' found. Unique values: 'glioblastoma patient'.
INFO:bulkdgd.recount3.util:sample attribute 'time_point' found. Unique values: 'before bevacizumab combination treatment', 'after bevacizumab combination treatment'.
INFO:bulkdgd.recount3.util:Experiments' attributes were found in the metadata (see below).
INFO:bulkdgd.recount3.util:experiment attribute 'GEO_Accession' found. Unique values: 'GSM2100916', 'GSM2100917', 'GSM2100918', 'GSM2100919', 'GSM2100920', 'GSM2100921', 'GSM2100922', 'GSM2100923', 'GSM2100924', 'GSM2100925', 'GSM2100926', 'GSM2100927', 'GSM2100928', 'GSM2100929', 'GSM2100930', 'GSM2100931', 'GSM2100932', 'GSM2100933', 'GSM2100934', 'GSM2100935', 'GSM2100936', 'GSM2100937', 'GSM2100938', 'GSM2100939', 'GSM2100940', 'GSM2100941', 'GSM2100942', 'GSM2100943', 'GSM2100944', 'GSM2100945', 'GSM2100946', 'GSM2100947', 'GSM2100948', 'GSM2100949', 'GSM2100950', 'GSM2100951'.
INFO:bulkdgd.recount3.util:The metadata with the sample/experiment attributes split into different columns were successfully written in 'recount3_raw/sra_SRP072494_metadata.gz'.
INFO:bulkdgd.recount3.util:The RNA-seq data were successfully saved in 'recount3_raw/sra_SRP072494_gene_sums.gz'.
INFO:bulkdgd.recount3.util:The QC metadata were successfully saved in 'recount3_raw/sra_SRP072494_qc.gz'.
INFO:bulkdgd.recount3.util:Retrieving metadata from: http://duffel.rail.bio/recount3/human/data_sources/sra/metadata/40/SRP141440/sra.sra.SRP141440.MD.gz.
INFO:bulkdgd.recount3.util:Samples' attributes were found in the metadata (see below).
INFO:bulkdgd.recount3.util:sample attribute 'age' found. Unique values: '29', '76', '62', '41', '57', '52', '34', '67', '64', '56', '73', '69', '65', '66', '44', '84', '58', '70', '79'.
INFO:bulkdgd.recount3.util:sample attribute 'gender' found. Unique values: 'female', 'male'.
INFO:bulkdgd.recount3.util:sample attribute 'source_name' found. Unique values: 'Glioblastoma'.
INFO:bulkdgd.recount3.util:sample attribute 'tissue' found. Unique values: 'high grade glioblastoma tumor'.
INFO:bulkdgd.recount3.util:Experiments' attributes were found in the metadata (see below).
INFO:bulkdgd.recount3.util:experiment attribute 'GEO_Accession' found. Unique values: 'GSM3106683', 'GSM3106684', 'GSM3106685', 'GSM3106686', 'GSM3106687', 'GSM3106688', 'GSM3106689', 'GSM3106690', 'GSM3106691', 'GSM3106692', 'GSM3106693', 'GSM3106694', 'GSM3106695', 'GSM3106696', 'GSM3106697', 'GSM3106698', 'GSM3106699', 'GSM3106700', 'GSM3106701', 'GSM3106702', 'GSM3106703', 'GSM3106704', 'GSM3106705', 'GSM3106706'.
INFO:bulkdgd.recount3.util:The metadata with the sample/experiment attributes split into different columns were successfully written in 'recount3_raw/sra_SRP141440_metadata.gz'.
INFO:bulkdgd.recount3.util:The RNA-seq data were successfully saved in 'recount3_raw/sra_SRP141440_gene_sums.gz'.
INFO:bulkdgd.recount3.util:The QC metadata were successfully saved in 'recount3_raw/sra_SRP141440_qc.gz'.
Combine the studies#
Concatenate the four studies into a single data frame, so that all downloaded samples can be preprocessed and used together.
[4]:
# Concatenate the four studies into a single data frame. Since the
# studies do not all report the same metadata fields, missing values
# are introduced where a field is absent for a given study - this does
# not affect the gene expression columns, which are present for all
# samples in all studies.
df_samples = pd.concat(dfs_studies, axis = 0)
Preprocess the samples#
Preprocess the combined samples so that their genes match the set of genes included in the bulkdgd model.
[5]:
# Preprocess the combined samples so that their genes match the set of
# genes included in the bulkdgd model (excluding genes not in the
# model and adding a count of 0 for genes in the model but missing
# from the downloaded data).
df_preproc, genes_excluded, genes_missing = \
ioutil.preprocess_samples(df_samples = df_samples)
INFO:bulkdgd.ioutil.samplesio:64837 column(s) containing gene expression data was (were) found in the input data frame.
INFO:bulkdgd.ioutil.samplesio:893 column(s) containing additional information (not gene expression data) was (were) in the input data frame: recount3_project_name, recount3_samples_category, rail_id, study, tcga_barcode, gdc_file_id, gdc_cases.project.name, gdc_cases.project.released, gdc_cases.project.state, gdc_cases.project.primary_site, gdc_cases.project.project_id, cgc_sample_sample_type, gdc_cases.case_id, gdc_updated_datetime, gdc_file_name, gdc_submitter_id, gdc_file_size, gdc_access, gdc_platform, gdc_state, gdc_type, gdc_file_state, gdc_experimental_strategy, gdc_data_type, gdc_md5sum, gdc_data_format, gdc_data_category, gdc_center.code, gdc_center.name, gdc_center.short_name, gdc_center.center_id, gdc_center.namespace, gdc_center.center_type, gdc_metadata_files.updated_datetime.analysis, gdc_metadata_files.created_datetime.analysis, gdc_metadata_files.file_name.analysis, gdc_metadata_files.md5sum.analysis, gdc_metadata_files.data_format.analysis, gdc_metadata_files.access.analysis, gdc_metadata_files.file_id.analysis, gdc_metadata_files.data_category.analysis, gdc_metadata_files.file_size.analysis, gdc_metadata_files.file_state.analysis, gdc_metadata_files.updated_datetime.experiment, gdc_metadata_files.file_name.experiment, gdc_metadata_files.md5sum.experiment, gdc_metadata_files.file_id.experiment, gdc_metadata_files.file_size.experiment, gdc_metadata_files.updated_datetime.run, gdc_metadata_files.file_name.run, gdc_metadata_files.md5sum.run, gdc_metadata_files.file_id.run, gdc_metadata_files.file_size.run, gdc_cases.updated_datetime, gdc_cases.submitter_id, gdc_cases.demographic.updated_datetime, gdc_cases.demographic.gender, gdc_cases.demographic.submitter_id, gdc_cases.demographic.year_of_birth, gdc_cases.demographic.race, gdc_cases.demographic.demographic_id, gdc_cases.demographic.ethnicity, gdc_cases.demographic.year_of_death, gdc_cases.project.program.dbgap_accession_number, gdc_cases.project.program.name, gdc_cases.project.program.program_id, gdc_cases.tissue_source_site.project, gdc_cases.tissue_source_site.bcr_id, gdc_cases.tissue_source_site.code, gdc_cases.tissue_source_site.tissue_source_site_id, gdc_cases.tissue_source_site.name, gdc_cases.diagnoses.classification_of_tumor, gdc_cases.diagnoses.primary_diagnosis, gdc_cases.diagnoses.submitter_id, gdc_cases.diagnoses.tumor_stage, gdc_cases.diagnoses.age_at_diagnosis, gdc_cases.diagnoses.vital_status, gdc_cases.diagnoses.morphology, gdc_cases.diagnoses.days_to_death, gdc_cases.diagnoses.days_to_last_follow_up, gdc_cases.diagnoses.diagnosis_id, gdc_cases.diagnoses.tissue_or_organ_of_origin, gdc_cases.diagnoses.days_to_birth, gdc_cases.diagnoses.treatments.submitter_id, gdc_cases.diagnoses.treatments.treatment_id, gdc_cases.samples.sample_type_id, gdc_cases.samples.oct_embedded, gdc_cases.samples.submitter_id, gdc_cases.samples.intermediate_dimension, gdc_cases.samples.sample_id, gdc_cases.samples.is_ffpe, gdc_cases.samples.pathology_report_uuid, gdc_cases.samples.sample_type, gdc_cases.samples.shortest_dimension, gdc_cases.samples.days_to_collection, gdc_cases.samples.initial_weight, gdc_cases.samples.longest_dimension, gdc_cases.samples.portions.creation_datetime, gdc_cases.samples.portions.portion_number, gdc_cases.samples.portions.weight, gdc_cases.samples.portions.submitter_id, gdc_cases.samples.portions.portion_id, gdc_cases.samples.portions.analytes.analyte_type, gdc_cases.samples.portions.analytes.submitter_id, gdc_cases.samples.portions.analytes.analyte_id, gdc_cases.samples.portions.analytes.a260_a280_ratio, gdc_cases.samples.portions.analytes.concentration, gdc_cases.samples.portions.analytes.spectrophotometer_method, gdc_cases.samples.portions.analytes.analyte_type_id, gdc_cases.samples.portions.analytes.aliquots.source_center, gdc_cases.samples.portions.analytes.aliquots.submitter_id, gdc_cases.samples.portions.analytes.aliquots.aliquot_id, gdc_cases.samples.portions.analytes.aliquots.concentration, gdc_cases.samples.portions.annotations.category, gdc_cases.samples.portions.annotations.status, gdc_cases.samples.portions.annotations.classification, gdc_cases.samples.portions.annotations.updated_datetime, gdc_cases.samples.portions.annotations.created_datetime, gdc_cases.samples.portions.annotations.notes, gdc_cases.samples.portions.annotations.submitter_id, gdc_cases.samples.portions.annotations.annotation_id, gdc_cases.samples.annotations.category, gdc_cases.samples.annotations.status, gdc_cases.samples.annotations.classification, gdc_cases.samples.annotations.updated_datetime, gdc_cases.samples.annotations.created_datetime, gdc_cases.samples.annotations.notes, gdc_cases.samples.annotations.submitter_id, gdc_cases.samples.annotations.annotation_id, gdc_cases.exposures.cigarettes_per_day, gdc_cases.exposures.weight, gdc_cases.exposures.alcohol_history, gdc_cases.exposures.bmi, gdc_cases.exposures.years_smoked, gdc_cases.exposures.height, gdc_cases.exposures.exposure_id, gdc_cases.exposures.submitter_id, gdc_associated_entities.entity_type, cgc_filename, cgc_file_upload_date, cgc_file_last_modified_date, cgc_file_submitter_id, cgc_file_sample, cgc_file_data_subtype, cgc_file_size, cgc_file_access_level, cgc_file_platform, cgc_file_investigation, cgc_file_experimental_strategy, cgc_file_data_type, cgc_file_aliquot, cgc_file_data_submitting_center, cgc_file_file_id, cgc_file_case, cgc_file_disease_type, cgc_file_data_format, cgc_file_portion, cgc_file_published_date, cgc_file_analyte, cgc_sample_country_of_sample_procurement, cgc_sample_is_ffpe, cgc_sample_sample_type_code, cgc_sample_shortest_dimension, cgc_sample_tissue_source_site, cgc_sample_days_to_collection, cgc_sample_portion, cgc_sample_intermediate_dimension, cgc_sample_tissue_source_site_code, cgc_sample_file, cgc_sample_initial_weight, cgc_sample_id, cgc_sample_longest_dimension, cgc_case_drug_therapy, cgc_case_program, cgc_case_year_of_diagnosis, cgc_case_new_tumor_event, cgc_case_primary_therapy_outcome_success, cgc_case_tumor_status, cgc_case_sample, cgc_case_age_at_diagnosis, cgc_case_file, cgc_case_vital_status, cgc_case_days_to_death, cgc_case_id, cgc_case_ethnicity, cgc_case_follow_up, cgc_case_new_tumor_event_after_initial_treatment, cgc_case_days_to_last_follow_up, cgc_case_prior_diagnosis, cgc_case_primary_site, cgc_case_icd_o3site, cgc_case_pathologic_n, cgc_case_histological_diagnosis, cgc_case_performance_status_score_ecog, cgc_case_performance_status_score_timing, cgc_case_clinical_stage, cgc_case_pathologic_stage, cgc_case_performance_status_score_karnofsky, cgc_case_pathologic_t, cgc_case_icd_o3histology, cgc_case_other_histological_diagnosis, cgc_case_gender, cgc_case_icd10, cgc_case_batch_number, cgc_case_clinical_m, cgc_case_clinical_n, cgc_case_radiation_therapy, cgc_case_race, cgc_case_clinical_t, cgc_portion_portion_number, cgc_portion_weight, cgc_portion_slide, cgc_portion_analyte, cgc_portion_file, cgc_portion_id, cgc_slide_percent_tumor_nuclei, cgc_slide_percent_normal_cells, cgc_slide_percent_monocyte_infiltration, cgc_slide_percent_lymphocyte_infiltration, cgc_slide_percent_stromal_cells, cgc_slide_section_location, cgc_slide_percent_necrosis, cgc_slide_percent_neutrophil_infiltration, cgc_slide_id, cgc_slide_percent_tumor_cells, cgc_drug_therapy_drug_name, cgc_drug_therapy_pharmaceutical_therapy_type, cgc_drug_therapy_id, cgc_radiation_therapy_id, cgc_radiation_therapy_radiation_type, cgc_radiation_therapy_radiation_therapy_site, cgc_follow_up_new_tumor_event, cgc_follow_up_new_tumor_event_after_initial_treatment, cgc_follow_up_days_to_last_follow_up, cgc_follow_up_tumor_status, cgc_follow_up_vital_status, cgc_follow_up_days_to_death, cgc_follow_up_id, xml_bcr_patient_barcode, xml_additional_studies, xml_tissue_source_site, xml_patient_id, xml_informed_consent_verified, xml_icd_o_3_site, xml_icd_o_3_histology, xml_icd_10, xml_tissue_prospective_collection_indicator, xml_tissue_retrospective_collection_indicator, xml_days_to_birth, xml_race_list, xml_ethnicity, xml_other_dx, xml_history_of_neoadjuvant_treatment, xml_vital_status, xml_days_to_last_followup, xml_days_to_death, xml_radiation_therapy, xml_postoperative_rx_tx, xml_person_neoplasm_cancer_status, xml_has_new_tumor_events_information, xml_day_of_form_completion, xml_month_of_form_completion, xml_year_of_form_completion, xml_has_follow_ups_information, xml_has_drugs_information, xml_has_radiations_information, xml_primary_pathology_histological_type, xml_primary_pathology_leiomyosarcoma_histologic_subtype, xml_primary_pathology_primary_tumor_lower_uterus_segment, xml_primary_pathology_leiomyosarcoma_major_vessel_involvement, xml_primary_pathology_synovial_sarcoma, xml_primary_pathology_tumor_depth, xml_primary_pathology_tumor_tissue_sites, xml_primary_pathology_tumor_tissue_site_other, xml_primary_pathology_days_to_initial_pathologic_diagnosis, xml_primary_pathology_age_at_initial_pathologic_diagnosis, xml_primary_pathology_year_of_initial_pathologic_diagnosis, xml_primary_pathology_margin_status, xml_primary_pathology_residual_tumor, xml_primary_pathology_tumor_total_necrosis_percent, xml_primary_pathology_tumor_necrosis_percent, xml_primary_pathology_mitotic_count, xml_primary_pathology_tumor_multifocal, xml_primary_pathology_discontiguous_lesion_count, xml_primary_pathology_tumor_sizes, xml_primary_pathology_radiologic_tumor_burden, xml_primary_pathology_pathologic_tumor_burden, xml_primary_pathology_local_disease_recurrence, xml_primary_pathology_metastatic_diagnosis, xml_primary_pathology_metastatic_site_at_diagnosis, xml_primary_pathology_metastatic_site_at_diagnosis_other, xml_primary_pathology_contiguous_organ_resection_site, xml_primary_pathology_other_contiguous_organ_resection_site, xml_primary_pathology_contiguous_organ_invaded, xml_primary_pathology_well_differentiated_liposarcoma_primary_dx, xml_primary_pathology_days_to_well_differentiated_liposarcoma_primary_dx, xml_primary_pathology_days_to_well_differentiated_liposarcoma_resection, xml_tumor_tissue_site, xml_adenocarcinoma_invasion, xml_histological_type, xml_histological_type_other, xml_tumor_type, xml_anatomic_neoplasm_subdivision, xml_anatomic_neoplasm_subdivision_other, xml_days_to_initial_pathologic_diagnosis, xml_age_at_initial_pathologic_diagnosis, xml_year_of_initial_pathologic_diagnosis, xml_initial_pathologic_diagnosis_method, xml_init_pathology_dx_method_other, xml_surgery_performed_type, xml_primary_lymph_node_presentation_assessment, xml_lymph_node_examined_count, xml_number_of_lymphnodes_positive_by_he, xml_number_of_lymphnodes_positive_by_ihc, xml_neoplasm_histologic_grade, xml_histologic_grading_tier_category, xml_maximum_tumor_dimension, xml_residual_tumor, xml_patient_death_reason, xml_source_of_patient_death_reason, xml_tobacco_smoking_history, xml_year_of_tobacco_smoking_onset, xml_stopped_smoking_year, xml_number_pack_years_smoked, xml_alcoholic_exposure_category, xml_frequency_of_alcohol_consumption, xml_amount_of_alcohol_consumption_per_day, xml_history_of_diabetes, xml_days_to_diabetes_onset, xml_history_of_chronic_pancreatitis, xml_days_to_pancreatitis_onset, xml_family_history_of_cancer, xml_relative_cancer_types, xml_targeted_molecular_therapy, xml_history_prior_surgery_type_other, xml_primary_therapy_outcome_success, xml_stage_event_system_version, xml_stage_event_clinical_stage, xml_stage_event_pathologic_stage, xml_stage_event_tnm_categories, xml_stage_event_psa, xml_stage_event_gleason_grading, xml_stage_event_ann_arbor, xml_stage_event_serum_markers, xml_stage_event_igcccg_stage, xml_stage_event_masaoka_stage, xml_days_to_last_known_alive, xml_preoperative_pretreatment_cea_level, xml_non_nodal_tumor_deposits, xml_circumferential_resection_margin, xml_venous_invasion, xml_lymphatic_invasion, xml_perineural_invasion_present, xml_microsatellite_instability, xml_number_of_loci_tested, xml_number_of_abnormal_loci, xml_kras_gene_analysis_performed, xml_kras_mutation_found, xml_kras_mutation_codon, xml_braf_gene_analysis_performed, xml_braf_gene_analysis_result, xml_synchronous_colon_cancer_present, xml_history_of_colon_polyps, xml_colon_polyps_present, xml_loss_expression_of_mismatch_repair_proteins_by_ihc, xml_loss_expression_of_mismatch_repair_proteins_by_ihc_results, xml_number_of_first_degree_relatives_with_cancer_diagnosis, xml_pleurodesis_performed, xml_pleurodesis_performed_90_days, xml_history_asbestos_exposure, xml_asbestos_exposure_type, xml_asbestos_exposure_source, xml_asbestos_exposure_age, xml_asbestos_exposure_years, xml_asbestos_exposure_age_last, xml_primary_occupation, xml_primary_occupation_other, xml_person_occupation_years_number, xml_blood_relative_cancer_history_list, xml_karnofsky_performance_score, xml_eastern_cancer_oncology_group, xml_performance_status_scale_timing, xml_primary_pathology_tumor_tissue_site, xml_primary_pathology_laterality, xml_primary_pathology_serum_mesothelin_prior_tx, xml_primary_pathology_serum_mesothelin_lower_limit, xml_primary_pathology_serum_mesothelin_upper_limit, xml_primary_pathology_creatinine_prior_tx, xml_primary_pathology_creatinine_norm_range_lower, xml_primary_pathology_creatinine_norm_range_upper, xml_primary_pathology_suv_of_pleura_max, xml_primary_pathology_mesothelioma_detection_method, xml_prior_glioma, xml_laterality, xml_lactate_dehydrogenase_result, xml_serum_calcium_result, xml_hemoglobin_result, xml_platelet_qualitative_result, xml_white_cell_count_result, xml_erythrocyte_sedimentation_rate_result, xml_number_of_lymphnodes_positive, xml_mitotane_therapy, xml_mitotane_therapy_adjuvant_setting, xml_therapeutic_mitotane_levels_achieved, xml_therapeutic_mitotane_lvl_recurrence, xml_mitotane_therapy_for_macroscopic_residual_disease, xml_therapeutic_mitotane_lvl_macroscopic_residual, xml_therapeutic_mitotane_lvl_progression, xml_post_surgical_procedure_assessment_thyroid_gland_carcinoma_status, xml_primary_pathology_ct_scan, xml_primary_pathology_ct_scan_findings, xml_primary_pathology_primary_lymph_node_presentation_assessment, xml_primary_pathology_lymph_node_examined_count, xml_primary_pathology_number_of_lymphnodes_positive_by_he, xml_primary_pathology_weiss_assessment_report, xml_primary_pathology_mitoses_count, xml_primary_pathology_metastatic_neoplasm_confirmed_diagnosis_method_names, xml_primary_pathology_metastatic_neoplasm_initial_diagnosis_anatomic_sites, xml_primary_pathology_excess_adrenal_hormone_history_types, xml_primary_pathology_germline_genotype_testing_report, xml_menopause_status, xml_birth_control_pill_history_usage_category, xml_number_of_successful_pregnancies_which_resulted_in_at_least_1_live_birth, xml_patient_pregnancy_spontaneous_abortion_count, xml_patient_pregnancy_therapeutic_abortion_count, xml_ectopic_pregnancy_count, xml_pregnancy_stillbirth_count, xml_female_breast_feeding_or_pregnancy_status_indicator, xml_total_number_of_pregnancies, xml_age_began_smoking_in_years, xml_patient_history_immune_system_and_related_disorders_names, xml_patient_history_immune_system_and_related_disorders_text, xml_assessment_timepoint_category, xml_days_to_performance_status_assessment, xml_tumor_response_cdus_type, xml_keratinizing_squamous_cell_carcinoma_present_indicator, xml_lymph_node_location_positive_pathology_names, xml_lymph_node_location_positive_pathology_text, xml_hysterectomy_performed_type, xml_hysterectomy_performed_text, xml_cervical_neoplasm_pathologic_margin_involved_type, xml_cervical_neoplasm_pathologic_margin_involved_text, xml_cervical_carcinoma_pelvic_extension_text, xml_lymphovascular_invasion_indicator, xml_cervical_carcinoma_corpus_uteri_involvement_indicator, xml_days_to_fdg_or_ct_pet_performed, xml_standardized_uptake_value_cervix_uteri_assessment_measurement, xml_fdg_or_ct_pet_performed_outcomes, xml_days_to_diagnostic_mri_performed, xml_diagnostic_mri_result_outcomes, xml_days_to_diagnostic_computed_tomography_performed, xml_diagnostic_ct_result_outcomes, xml_human_papillomavirus_types, xml_human_papillomavirus_other_type_text, xml_human_papillomavirus_laboratory_procedure_performed_name, xml_human_papillomavirus_laboratory_procedure_performed_text, xml_oligonucleotide_primer_pair_laboratory_procedure_performed_name, xml_pregnant_at_diagnosis, xml_treatment, xml_er_detection_method_text, xml_pgr_detection_method_text, xml_anatomic_neoplasm_subdivisions, xml_her2_neu_chromosone_17_signal_ratio_value, xml_axillary_lymph_node_stage_method_type, xml_axillary_lymph_node_stage_other_method_descriptive_text, xml_breast_carcinoma_surgical_procedure_name, xml_breast_neoplasm_other_surgical_procedure_descriptive_text, xml_breast_carcinoma_primary_surgical_procedure_name, xml_surgical_procedure_purpose_other_text, xml_breast_carcinoma_progesterone_receptor_status, xml_cytokeratin_immunohistochemistry_staining_method_micrometastasis_indicator, xml_breast_carcinoma_immunohistochemistry_er_pos_finding_scale, xml_immunohistochemistry_positive_cell_score, xml_her2_immunohistochemistry_level_result, xml_breast_cancer_surgery_margin_status, xml_margin_status, xml_lab_procedure_her2_neu_in_situ_hybrid_outcome_type, xml_breast_carcinoma_estrogen_receptor_status, xml_lab_proc_her2_neu_immunohistochemistry_receptor_status, xml_pos_finding_progesterone_receptor_other_measurement_scale_text, xml_positive_finding_estrogen_receptor_other_measurement_scale_text, xml_her2_erbb_pos_finding_cell_percent_category, xml_pos_finding_her2_erbb2_other_measurement_scale_text, xml_her2_erbb_method_calculation_method_text, xml_her2_neu_and_centromere_17_copy_number_analysis_input_total_number_count, xml_her2_and_centromere_17_positive_finding_other_measurement_scale_text, xml_her2_erbb_pos_finding_fluorescence_in_situ_hybridization_calculation_method_text, xml_fluorescence_in_situ_hybridization_diagnostic_procedure_chromosome_17_signal_result_range, xml_first_nonlymph_node_metastasis_anatomic_sites, xml_er_level_cell_percentage_category, xml_progesterone_receptor_level_cell_percent_category, xml_distant_metastasis_present_ind2, xml_metastatic_breast_carcinoma_estrogen_receptor_status, xml_metastatic_breast_carcinoma_estrogen_receptor_level_cell_percent_category, xml_metastatic_breast_carcinoma_immunohistochemistry_er_pos_cell_score, xml_metastatic_breast_carcinoma_progesterone_receptor_status, xml_metastatic_breast_carcinoma_lab_proc_her2_neu_immunohistochemistry_receptor_status, xml_metastatic_breast_carcinoma_progesterone_receptor_level_cell_percent_category, xml_metastatic_breast_carcinoma_immunohistochemistry_pr_pos_cell_score, xml_metastatic_breast_carcinoma_her2_erbb_pos_finding_cell_percent_category, xml_metastatic_breast_carcinoma_erbb2_immunohistochemistry_level_result, xml_metastatic_breast_carcinoma_lab_proc_her2_neu_in_situ_hybridization_outcome_type, xml_metastatic_breast_carcinoma_her2_neu_chromosone_17_signal_ratio_value, xml_her2_neu_breast_carcinoma_copy_analysis_input_total_number, xml_breast_carcinoma_immunohistochemistry_progesterone_receptor_pos_finding_scale, xml_breast_carcinoma_immunohistochemistry_pos_cell_score, xml_country_of_birth, xml_country_of_procurement, xml_state_province_of_procurement, xml_city_of_procurement, xml_reflux_history, xml_antireflux_treatment_types, xml_h_pylori_infection, xml_initial_diagnosis_by, xml_barretts_esophagus, xml_goblet_cells_present, xml_history_of_esophageal_cancer, xml_number_of_relatives_diagnosed, xml_primary_pathology_esophageal_tumor_cental_location, xml_primary_pathology_esophageal_tumor_involvement_sites, xml_primary_pathology_columnar_metaplasia_present, xml_primary_pathology_columnar_mucosa_goblet_cell_present, xml_primary_pathology_columnar_mucosa_dysplasia, xml_primary_pathology_neoplasm_histologic_grade, xml_primary_pathology_initial_pathologic_diagnosis_method, xml_primary_pathology_init_pathology_dx_method_other, xml_primary_pathology_lymph_node_metastasis_radiographic_evidence, xml_primary_pathology_number_of_lymphnodes_positive_by_ihc, xml_primary_pathology_planned_surgery_status, xml_primary_pathology_treatment_prior_to_surgery, xml_primary_pathology_karnofsky_performance_score, xml_primary_pathology_eastern_cancer_oncology_group, xml_primary_pathology_radiation_therapy, xml_primary_pathology_postoperative_rx_tx, xml_history_of_disease, xml_tests_performed, xml_primary_pathology_follicular_percent, xml_primary_pathology_lymph_node_involvement_sites, xml_primary_pathology_extranodal_involvment_site_other, xml_primary_pathology_extranodal_sites_involvement_number, xml_primary_pathology_maximum_tumor_dimension, xml_primary_pathology_maximum_tumor_bulk_anatomic_site, xml_primary_pathology_bone_marrow_biopsy_done, xml_primary_pathology_bone_marrow_involvement, xml_primary_pathology_bone_marrow_sample_histology, xml_presence_of_sarcomatoid_features, xml_percent_tumor_sarcomatoid, xml_history_pheo_or_para_include_benign, xml_history_pheo_or_para_anatomic_site, xml_primary_pathology_outside_adrenal, xml_primary_pathology_disease_detected_on_screening, xml_eye_color, xml_primary_pathology_tumor_tissue_site_list, xml_primary_pathology_tumor_morphology_list, xml_primary_pathology_cytogenetic_abnormality_list, xml_primary_pathology_gene_expression_profile_list, xml_primary_pathology_extravascular_matrix_patterns, xml_primary_pathology_tumor_infiltrating_lymphocytes, xml_primary_pathology_tumor_infiltrating_macrophages, xml_primary_pathology_tumor_basal_diameter, xml_primary_pathology_tumor_basal_diameter_mx, xml_primary_pathology_tumor_thickness, xml_primary_pathology_tumor_thickness_measurement, xml_primary_pathology_extrascleral_extension, xml_primary_pathology_extranocular_nodule_size, xml_primary_pathology_tumor_shape_pathologic_clinical, xml_primary_pathology_metastatic_site_list, xml_primary_pathology_other_metastatic_site, xml_primary_neoplasm_melanoma_dx, xml_sites_of_primary_melanomas, xml_breslow_depth_value, xml_melanoma_clark_level_value, xml_melanoma_ulceration_indicator, xml_malignant_neoplasm_mitotic_count_rate, xml_days_to_submitted_specimen_dx, xml_distant_metastasis_anatomic_site, xml_melanoma_origin_skin_anatomic_site, xml_prior_systemic_therapy_types, xml_interferon_90_day_prior_excision_admin_indicator, xml_leukemia_specimen_cell_source_type, xml_cumulative_agent_total_dose, xml_prior_hematologic_disorder_diagnosis_indicator, xml_total_dose_units, xml_hydroxyurea_agent_administered_day_count, xml_hydroxyurea_administration_prior_registration_clinical_study_indicator, xml_atra_exposure, xml_steroid_therapy_administered, xml_person_history_nonmedical_leukemia_causing_agent_type, xml_lab_procedure_blast_cell_outcome_percentage_value, xml_leukemia_french_american_british_morphology_code, xml_immunophenotype_cytochemistry_testing_results, xml_lab_procedure_bone_marrow_cellularity_outcome_percent_value, xml_lab_procedure_leukocyte_result_unspecified_value, xml_lab_procedure_hemoglobin_result_specified_value, xml_lab_procedure_hematocrit_outcome_percent_value, xml_platelet_result_count, xml_lab_procedure_bone_marrow_blast_cell_outcome_percent_value, xml_lab_procedure_bone_marrow_promyelocyte_result_percent_value, xml_lab_procedure_bone_marrow_myelocyte_result_percent_value, xml_lab_procedure_bone_marrow_metamyelocyte_result_value, xml_lab_procedure_bone_marrow_band_cell_result_percent_value, xml_lab_procedure_bone_marrow_neutrophil_result_percent_value, xml_lab_procedure_bone_marrow_lab_eosinophil_result_percent_value, xml_lab_procedure_bone_marrow_basophil_result_percent_value, xml_lab_procedure_bone_marrow_lymphocyte_outcome_percent_value, xml_lab_procedure_monocyte_result_percent_value, xml_lab_procedure_bone_marrow_prolymphocyte_result_percent_value, xml_lab_procedure_bone_marrow_promonocyte_count_result_percent_value, xml_lab_procedure_abnormal_lymphocyte_result_percent_value, xml_cytogenetic_analysis_performed_ind, xml_fluorescence_in_situ_hybrid_cytogenetics_metaphase_nucleus_result_count, xml_acute_myeloid_leukemia_calgb_cytogenetics_risk_category, xml_cytogenetic_abnormalities, xml_cytogenetic_abnormality_other, xml_fish_evaluation_performed_ind, xml_fluorescence_in_situ_hybridization_abnormal_result_indicator, xml_fish_test_component_results, xml_germline_testing_performed, xml_disease_detection_molecular_analysis_method_type, xml_molecular_analysis_abnormal_result_indicator, xml_molecular_analysis_abnormality_testing_results, xml_horm_ther, xml_prior_tamoxifen_administered_usage_category, xml_hypertension, xml_diabetes, xml_pregnancies, xml_colorectal_cancer, xml_primary_pathology_anatomic_neoplasm_subdivision, xml_primary_pathology_surgical_approach, xml_primary_pathology_peritoneal_wash, xml_primary_pathology_pct_tumor_invasion, xml_primary_pathology_total_pelv_lnr, xml_primary_pathology_pln_pos_light_micro, xml_primary_pathology_pln_pos_ihc, xml_primary_pathology_total_pelv_lnp, xml_primary_pathology_total_aor_lnr, xml_primary_pathology_aln_pos_light_micro, xml_primary_pathology_aln_pos_ihc, xml_primary_pathology_total_aor.lnp, xml_diagnosis, xml_location_in_lung_parenchyma, xml_kras_mutation_result, xml_egfr_mutation_performed, xml_egfr_mutation_result, xml_eml4_alk_translocation_performed, xml_eml4_alk_translocation_method, xml_pulmonary_function_test_performed, xml_pre_bronchodilator_fev1_percent, xml_post_bronchodilator_fev1_percent, xml_pre_bronchodilator_fev1_fvc_percent, xml_post_bronchodilator_fev1_fvc_percent, xml_dlco_predictive_percent, xml_lymphnode_neck_dissection, xml_lymphnode_dissection_method_left, xml_lymphnode_dissection_method_right, xml_egfr_amplication_status, xml_presence_of_pathological_nodal_extracapsular_spread, xml_lymphovascular_invasion_present, xml_hpv_status_by_p16_testing, xml_hpv_status_by_ish_testing, xml_patient_personal_medical_history_thyroid_gland_disorder_names, xml_patient_personal_medical_history_thyroid_other_specify_text, xml_first_degree_relative_history_thyroid_gland_carcinoma_diagnosis_relationship_types, xml_person_lifetime_risk_radiation_exposure_indicator, xml_primary_thyroid_gland_neoplasm_location_anatomic_site, xml_primary_neoplasm_focus_type, xml_neoplasm_dimension, xml_lymph_node_preoperative_scan_indicator, xml_lymph_node_preoperative_assessment_diagnostic_imaging_types, xml_extrathyroid_carcinoma_present_extension_status, xml_metastatic_neoplasm_confirmed_diagnosis_method_names, xml_metastatic_neoplasm_confirmed_diagnosis_method_text, xml_metastatic_site, xml_other_metastatic_site, xml_genotype_analysis_performed_indicator, xml_genotyping_results_gene_mutation_not_reported_reasons, xml_braf_gene_genotyping_outcome_lab_results_text, xml_ras_family_gene_genotyping_outcome_lab_results_text, xml_ret_ptc_rearrangement_genotyping_outcome_lab_results_text, xml_other_genotyping_outcome_lab_results_text, xml_i_131_total_administered_preparation_technique, xml_i_131_first_administered_dose, xml_i_131_subsequent_administered_dose, xml_i_131_total_administered_dose, xml_radiation_therapy_administered_preparation_technique_text, xml_radiation_therapy_administered_dose_text, xml_radiosensitizing_agent_administered_indicator, xml_person_occupation_description_text, xml_occupation_primary_job, xml_chemical_exposure_text, xml_person_primary_industry_text, xml_diagnosis_subtype, xml_disease_extracapsular_extension_ind.3, xml_bladder_carcinoma_extracapsular_extension_status, xml_metastatic_site_list, xml_person_concomitant_prostate_carcinoma_occurrence_indicator, xml_person_concomitant_prostate_carcinoma_pathologic_t_stage, xml_history_non_muscle_invasive_blca, xml_history_ionizing_rt_to_head, xml_tumor_location, xml_supratentorial_localization, xml_seizure_history, xml_headache_history, xml_mental_status_changes, xml_visual_changes, xml_sensory_changes, xml_motor_movement_changes, xml_first_presenting_symptom, xml_first_presenting_symptom_longest_duration, xml_asthma_history, xml_eczema_history, xml_hay_fever_history, xml_mold_or_dust_allergy_history, xml_first_diagnosis_age_asth_ecz_hay_fev_mold_dust, xml_food_allergy_history, xml_food_allergy_types, xml_first_diagnosis_age_of_food_allergy, xml_animal_insect_allergy_history, xml_animal_insect_allergy_types, xml_first_diagnosis_age_of_animal_insect_allergy, xml_preoperative_corticosteroids, xml_preoperative_antiseizure_meds, xml_family_history_of_primary_brain_tumor, xml_ldh1_mutation_tested, xml_ldh1_mutation_test_method, xml_ldh1_mutation_found, xml_inherited_genetic_syndrome_found, xml_inherited_genetic_syndrome_result, xml_history_of_undescended_testis, xml_level_of_non_descent, xml_undescended_testis_corrected, xml_undescended_testis_corrected_age, xml_undescended_testis_method_left, xml_undescended_testis_method_right, xml_history_hypospadias, xml_history_fertility, xml_family_history_testicular_cancer, xml_relation_testicular_cancer_list, xml_family_history_other_cancer, xml_postoperative_tx_list, xml_primary_pathology_bilateral_diagnosis_timing_type, xml_primary_pathology_days_to_bilateral_tumor_dx, xml_primary_pathology_synchronous_tumor_histology_list, xml_primary_pathology_testis_tumor_macroextent, xml_primary_pathology_testis_tumor_macroextent_other, xml_primary_pathology_testis_tumor_microextent, xml_primary_pathology_histology_list, xml_primary_pathology_intratubular_germ_cell_neoplasm, xml_primary_pathology_lymphovascular_invasion_present, xml_primary_pathology_pre_orchi_serum_marker_info, xml_primary_pathology_post_orchi_serum_marker_info, xml_primary_pathology_post_orchi_lymph_node_dissection, xml_primary_pathology_molecular_test_result_list, xml_primary_pathology_first_treatment_success, xml_surgical_approach, xml_peritoneal_wash, xml_pct_tumor_invasion, xml_total_pelv_lnr, xml_pln_pos_light_micro, xml_pln_pos_ihc, xml_total_pelv_lnp, xml_total_aor_lnr, xml_aln_pos_light_micro, xml_aln_pos_ihc, xml_total_aor.lnp, xml_relative_family_cancer_history, xml_cancer_first_degree_relative, xml_history_hepato_carcinoma_risk_factors, xml_post_op_ablation_embolization_tx, xml_specimen_collection_method_name, xml_vascular_tumor_cell_type, xml_child_pugh_classification_grade, xml_fetoprotein_outcome_value, xml_fetoprotein_outcome_lower_limit, xml_fetoprotein_outcome_upper_limit, xml_platelet_result_lower_limit, xml_platelet_result_upper_limit, xml_prothrombin_time_result_value, xml_inter_norm_ratio_lower_limit, xml_intern_norm_ratio_upper_limit, xml_albumin_result_specified_value, xml_albumin_result_lower_limit, xml_albumin_result_upper_limit, xml_bilirubin_upper_limit, xml_bilirubin_lower_limit, xml_total_bilirubin_upper_limit, xml_creatinine_value_in_mg_dl, xml_creatinine_lower_level, xml_creatinine_upper_limit, xml_fibrosis_ishak_score, xml_adjacent_hepatic_tissue_inflammation_extent_type, xml_viral_hepatitis_serologies, xml_ablations, xml_tumor_residual_disease, xml_jewish_origin, xml_zone_of_origin, xml_tumor_levels, xml_days_to_bone_scan_performed, xml_bone_scan_results, xml_diagnostic_ct_abd_pelvis_performed, xml_diagnostic_ct_abd_pelvis_results, xml_diagnostic_mri_performed, xml_diagnostic_mri_results, xml_lymphnodes_examined, xml_number_of_lymphnodes_examined, xml_biochemical_recurrence, xml_days_to_first_biochemical_recurrence, xml_primary_pathology_specimen_collection_method_name, xml_primary_pathology_history_prior_surgery_type_other, xml_primary_pathology_vascular_tumor_cell_type, xml_primary_pathology_perineural_invasion_present, xml_primary_pathology_child_pugh_classification_grade, xml_primary_pathology_ca_19_9_level, xml_primary_pathology_ca_19_9_level_lower, xml_primary_pathology_ca_19_9_level_upper, xml_primary_pathology_fetoprotein_outcome_value, xml_primary_pathology_fetoprotein_outcome_lower_limit, xml_primary_pathology_fetoprotein_outcome_upper_limit, xml_primary_pathology_platelet_result_count, xml_primary_pathology_platelet_result_lower_limit, xml_primary_pathology_platelet_result_upper_limit, xml_primary_pathology_prothrombin_time_result_value, xml_primary_pathology_inter_norm_ratio_lower_limit, xml_primary_pathology_intern_norm_ratio_upper_limit, xml_primary_pathology_albumin_result_specified_value, xml_primary_pathology_albumin_result_lower_limit, xml_primary_pathology_albumin_result_upper_limit, xml_primary_pathology_bilirubin_upper_limit, xml_primary_pathology_bilirubin_lower_limit, xml_primary_pathology_total_bilirubin_upper_limit, xml_primary_pathology_creatinine_value_in_mg_dl, xml_primary_pathology_creatinine_lower_level, xml_primary_pathology_creatinine_upper_limit, xml_primary_pathology_fibrosis_ishak_score, xml_primary_pathology_cholangitis_tissue_evidence, xml_antireflux_treatment, xml_family_history_of_stomach_cancer, xml_number_of_relatives_with_stomach_cancer, xml_primary_pathology_histological_type_list, xml_primary_pathology_history_myasthenia_gravis, xml_primary_pathology_section_myasthenia_gravis, sample_acc, experiment_acc, submission_acc, submission_center, submission_lab, study_title, study_abstract, study_description, experiment_title, design_description, sample_description, library_name, library_strategy, library_source, library_selection, library_layout, paired_nominal_length, paired_nominal_stdev, library_construction_protocol, platform_model, spot_length, sample_name, sample_title, sample_bases, sample_spots, run_published, size, run_total_bases, run_total_spots, num_reads, num_spots, read_info, run_alias, run_center_name, run_broker_name, run_center, history, source_name, GEO_Accession, batch, diagnosis, first-line_therapy, gender, library_preperation_date, rna-extraction_date, sample_from_primary_gbm_diagnosis, subject_group, subject_id, subject_status, time_point, age, tissue.
INFO:bulkdgd.ioutil.samplesio:Now looking for duplicated samples...
INFO:bulkdgd.ioutil.samplesio:No duplicated samples were found.
INFO:bulkdgd.ioutil.samplesio:Now looking for missing values in the columns containing gene expression data...
INFO:bulkdgd.ioutil.samplesio:No missing values were found in the columns containing gene expression data.
INFO:bulkdgd.ioutil.samplesio:Now looking for duplicated genes...
INFO:bulkdgd.ioutil.samplesio:No duplicated genes were found.
INFO:bulkdgd.ioutil.samplesio:In the data frame containing the pre-processed samples, the columns containing gene expression data will be ordered according to the list of genes included in the DGD model (taken from '/home/lcb518/software/bulkdgd-2.0.0/bulkdgd/data/model/genes/genes.txt').
INFO:bulkdgd.ioutil.samplesio:In the data frame containing the pre-processed samples, the columns found in the input data frame which did not contain gene expression data, if any were present, will be appended as the last columns of the data frame and appear in the same order as they did in the input data frame.
WARNING:bulkdgd.ioutil.samplesio:49942 gene(s) found in the input samples is (are) not part of the set of genes included in the DGD model. It (they) will be removed from the data frame of preprocessed samples.
INFO:bulkdgd.ioutil.samplesio:All genes included in the DGD model were found in the input samples.
Save the outputs#
Save the full set of preprocessed samples (not tracked in the repository because of its size), plus a small, illustrative subset (five samples per study) that is checked in and can be used directly in :doc:Tutorial 1 <tutorial_1>.
[6]:
# Save the full set of preprocessed samples (all downloaded samples
# from all four studies) - this file is not tracked in the repository
# because of its size, but it is what a real analysis would use.
ioutil.save_samples(\
df = df_preproc,
csv_file = f"{raw_wd}/samples_preprocessed_full.csv",
sep = ",")
# For the tutorial, save only a small, illustrative subset of the
# preprocessed samples (the first five samples of each study), so that
# the file checked into the repository stays small while still
# containing samples from all four studies.
df_preproc_subset = \
df_preproc.groupby(\
["recount3_project_name", "recount3_samples_category"],
group_keys = False).head(5)
ioutil.save_samples(\
df = df_preproc_subset,
csv_file = "samples_preprocessed.csv",
sep = ",")
# Save the lists of genes excluded from/missing in the downloaded
# data (these are the same regardless of how many samples are kept).
with open("genes_excluded.txt", "w") as f:
f.write("\n".join(genes_excluded))
with open("genes_missing.txt", "w") as f:
f.write("\n".join(genes_missing))
Summary#
Report how many samples were downloaded for each study, and how the preprocessing step changed the number of genes.
[7]:
# Report how many samples were downloaded for each study, and how the
# preprocessing step changed the number of genes.
log.info(\
"Samples downloaded per study:\n" + \
df_samples.groupby(\
["recount3_project_name",
"recount3_samples_category"]).size().to_string())
log.info(f"Number of genes excluded: {len(genes_excluded)}.")
log.info(f"Number of genes missing (added with a count of 0): " \
f"{len(genes_missing)}.")
log.info(f"Number of samples in the tutorial's subset: " \
f"{len(df_preproc_subset)}.")
INFO:root:Samples downloaded per study:
recount3_project_name recount3_samples_category
SRA SRP027383 274
SRP072494 36
SRP141440 64
TCGA GBM 175
INFO:root:Number of genes excluded: 49942.
INFO:root:Number of genes missing (added with a count of 0): 0.
INFO:root:Number of samples in the tutorial's subset: 20.