Evaluating pyDeid¶
We show how to evaluate pyDeid on a custom dataset of Canadian admission notes, as described in the paper
pyDeid: An Improved, Fast, Flexible, and Generalizable Rules-based Approach for De-identification of Free-text Medical Records
As well as on the popular n2c2 benchmark dataset of American discharge notes, using the ETUDE engine.
Testing CSV output against a gold standard dataset with CSVEvaluator¶
Given a test dataset of clinical notes formatted in a csv such as that found in tests/test.csv, and a “gold standard” dataset with each note split by token, and annotated with the appropriate PII type, we can evaluate the performance of pyDeid using the CSVEvaluator class.
First we run pyDeid on tests/test.csv, outputting in csv format.
[1]:
from pyDeid import pyDeid
pyDeid(
original_file = "../../tests/test.csv",
note_varname = "note_text",
encounter_id_varname = "genc_id",
note_id_varname = "note_id"
)
Processing encounter 3, note Record 3: : 3it [00:01, 2.75it/s]
Diagnostics:
- chars/s = 149.41586216889525
- s/note = 0.36363832155863446
From here we use the annotated ground truth dataset.
We show how to create a ground truth CSV given raw notes in the form of ./tests/test.csv.
Begin by tokenizing the raw notes using
tokenize_csv().Manually annotate the notes for PHI.
Using
melt_annotations(), combine multi-token PHI into a single entry.
In particular dates and locations are usually multi-token, so we handle them specifically.
[2]:
from pyDeid.phi_types import tokenize_csv, melt_annotations
tokenize_csv(
input_file = "../../tests/test.csv",
output_file = "../../tests/test_tokenized.csv",
encounter_id_varname = "genc_id",
note_id_varname = "note_id",
note_text_varname = "note_text"
)
Then we annotate the file ./tests/test_tokenized.csv by adding the PHI type to the annotation column.
Once that is complete, we combine multi-token PHI. In this example file, dates and locations are split across multiple tokens.
[3]:
melt_annotations(
input_file = "../../tests/test_tokenized.csv",
output_file = "../../tests/ground_truth_processed.csv",
merge_annotations = ["d", "l"]
)
Now we have a ground truth dataset that we can use to compare against the output of pyDeid.
To do this we use the CSVEvaluator class.
[4]:
from pyDeid.phi_types.CSVEvaluator import CSVEvaluator
evaluator = CSVEvaluator()
precision, recall, f1 = evaluator.add_ground_truth_file("../../tests/ground_truth.csv", note_id_varname = "note_id")\
.add_result_file("../../tests/test__PHI.csv")\
.evaluate()
print(
f"""
Precision: {precision}
Recall: {recall}
F1: {f1}
"""
)
Precision: 1.0
Recall: 1.0
F1: 1.0
Using the ETUDE Engine¶
The ETUDE engine is a well established, standard tool for analyzing de-identification performance against various benchmark dataset formats, such as the n2c2 format.
In this section, we use the ETUDE engine to evaluate the performance of pyDeid on the n2c2 dataset. We begin by cloning the repository.
In order to run the evaluation on n2c2, pyDeid does require the ability to read and write from xml. This is available through the pyDeid_n2c2() function.
Ensure that the n2c2 dataset is saved to some directory such as ./tests/n2c2.
[ ]:
from pyDeid.n2c2 import pyDeid_n2c2
pyDeid_n2c2(
input_dir = "path/to/n2c2_test_data",
output_dir = "path/to/pydeid_n2c2_output",
)
Note that this is essentially ready to run the evaluation on, however there is a significant difference between how pyDeid recognizes names and how names are annotated in the n2c2 ground truth. In pyDeid, first and last names are considered separately, and so we must separate these annotations in the n2c2 ground truth using split_multi_word_tags().
[ ]:
from pyDeid import split_multi_word_tags
split_multi_word_tags(
input_dir = "path/to/n2c2_test_data",
output_dir = "path/to/n2c2_test_data_preprocessed"
)
Now we are ready to run the ETUDE engine.
In order to compare a given reference file with a given test file, the ETUDE engine uses a config for each of the reference file, and the tool’s output. We provide both of the relevant configs under the ./tests directory.
With this, we run the following command from the directory in which we cloned the ETUDE engine repository.
[6]:
!python H:/repos/GitHub/etude-engine/etude.py \
--reference-input "path/to/n2c2_test_data_preprocessed" \
--reference-config ../../tests/n2c2_pydeid.conf \
--test-input "path/to/pydeid_n2c2_output" \
--test-config ../../tests/pydeid_n2c2.conf \
--by-type --score-key "Parent"
exact TP FP TN FN
micro-average 8280.0 1153.0 0.0 2816.0
Address 116.0 4.0 0.0 420.0
Contact Information 83.0 48.0 0.0 135.0
Identifiers 81.0 33.0 0.0 536.0
Names 4031.0 845.0 0.0 714.0
Time 3969.0 223.0 0.0 1011.0
macro-average by type 8280.0 1153.0 0.0 2816.0
0%| | 0/514 [00:00<?, ?it/s]
0%| | 1/514 [00:01<09:57, 1.16s/it]
0%| | 2/514 [00:02<08:39, 1.02s/it]
1%| | 3/514 [00:02<08:17, 1.03it/s]
1%| | 4/514 [00:04<08:57, 1.05s/it]
1%| | 5/514 [00:05<08:39, 1.02s/it]
1%|1 | 6/514 [00:06<08:24, 1.01it/s]
1%|1 | 7/514 [00:07<08:15, 1.02it/s]
2%|1 | 8/514 [00:08<08:55, 1.06s/it]
2%|1 | 9/514 [00:09<08:42, 1.04s/it]
2%|1 | 10/514 [00:10<08:32, 1.02s/it]
2%|2 | 11/514 [00:11<08:25, 1.01s/it]
2%|2 | 12/514 [00:12<09:35, 1.15s/it]
3%|2 | 13/514 [00:13<09:14, 1.11s/it]
3%|2 | 14/514 [00:14<08:55, 1.07s/it]
3%|2 | 15/514 [00:15<08:47, 1.06s/it]
3%|3 | 16/514 [00:16<08:39, 1.04s/it]
3%|3 | 17/514 [00:17<08:35, 1.04s/it]
4%|3 | 18/514 [00:18<08:48, 1.06s/it]
4%|3 | 19/514 [00:19<08:51, 1.07s/it]
4%|3 | 20/514 [00:20<08:41, 1.06s/it]
4%|4 | 21/514 [00:22<08:40, 1.05s/it]
4%|4 | 22/514 [00:23<08:59, 1.10s/it]
4%|4 | 23/514 [00:24<08:47, 1.07s/it]
5%|4 | 24/514 [00:25<08:42, 1.07s/it]
5%|4 | 25/514 [00:26<08:36, 1.06s/it]
5%|5 | 26/514 [00:27<09:04, 1.12s/it]
5%|5 | 27/514 [00:28<09:15, 1.14s/it]
5%|5 | 28/514 [00:29<09:04, 1.12s/it]
6%|5 | 29/514 [00:30<08:39, 1.07s/it]
6%|5 | 30/514 [00:31<08:25, 1.04s/it]
6%|6 | 31/514 [00:32<08:11, 1.02s/it]
6%|6 | 32/514 [00:33<08:04, 1.00s/it]
6%|6 | 33/514 [00:35<08:58, 1.12s/it]
7%|6 | 34/514 [00:36<08:43, 1.09s/it]
7%|6 | 35/514 [00:37<08:34, 1.07s/it]
7%|7 | 36/514 [00:38<08:19, 1.05s/it]
7%|7 | 37/514 [00:39<08:14, 1.04s/it]
7%|7 | 38/514 [00:40<08:39, 1.09s/it]
8%|7 | 39/514 [00:41<08:20, 1.05s/it]
8%|7 | 40/514 [00:42<08:02, 1.02s/it]
8%|7 | 41/514 [00:43<07:53, 1.00s/it]
8%|8 | 42/514 [00:44<07:47, 1.01it/s]
8%|8 | 43/514 [00:45<07:51, 1.00s/it]
9%|8 | 44/514 [00:46<07:59, 1.02s/it]
9%|8 | 45/514 [00:47<08:41, 1.11s/it]
9%|8 | 46/514 [00:48<09:13, 1.18s/it]
9%|9 | 47/514 [00:50<09:22, 1.20s/it]
9%|9 | 48/514 [00:51<08:49, 1.14s/it]
10%|9 | 49/514 [00:52<09:00, 1.16s/it]
10%|9 | 50/514 [00:53<09:10, 1.19s/it]
10%|9 | 51/514 [00:54<09:00, 1.17s/it]
10%|# | 52/514 [00:55<08:36, 1.12s/it]
10%|# | 53/514 [00:56<08:34, 1.12s/it]
11%|# | 54/514 [00:58<08:39, 1.13s/it]
11%|# | 55/514 [00:59<08:32, 1.12s/it]
11%|# | 56/514 [01:00<08:23, 1.10s/it]
11%|#1 | 57/514 [01:01<08:35, 1.13s/it]
11%|#1 | 58/514 [01:02<08:28, 1.11s/it]
11%|#1 | 59/514 [01:03<08:08, 1.07s/it]
12%|#1 | 60/514 [01:04<08:28, 1.12s/it]
12%|#1 | 61/514 [01:05<08:31, 1.13s/it]
12%|#2 | 62/514 [01:07<08:36, 1.14s/it]
12%|#2 | 63/514 [01:08<08:27, 1.13s/it]
12%|#2 | 64/514 [01:09<08:51, 1.18s/it]
13%|#2 | 65/514 [01:10<08:47, 1.17s/it]
13%|#2 | 66/514 [01:11<08:56, 1.20s/it]
13%|#3 | 67/514 [01:13<09:23, 1.26s/it]
13%|#3 | 68/514 [01:14<08:47, 1.18s/it]
13%|#3 | 69/514 [01:16<10:28, 1.41s/it]
14%|#3 | 70/514 [01:17<10:27, 1.41s/it]
14%|#3 | 71/514 [01:19<11:19, 1.53s/it]
14%|#4 | 72/514 [01:21<12:25, 1.69s/it]
14%|#4 | 73/514 [01:22<11:18, 1.54s/it]
14%|#4 | 74/514 [01:23<10:19, 1.41s/it]
15%|#4 | 75/514 [01:25<09:59, 1.37s/it]
15%|#4 | 76/514 [01:26<09:13, 1.26s/it]
15%|#4 | 77/514 [01:26<08:32, 1.17s/it]
15%|#5 | 78/514 [01:28<08:10, 1.13s/it]
15%|#5 | 79/514 [01:29<07:52, 1.09s/it]
16%|#5 | 80/514 [01:30<08:36, 1.19s/it]
16%|#5 | 81/514 [01:31<08:11, 1.14s/it]
16%|#5 | 82/514 [01:32<07:57, 1.11s/it]
16%|#6 | 83/514 [01:33<07:38, 1.06s/it]
16%|#6 | 84/514 [01:34<07:28, 1.04s/it]
17%|#6 | 85/514 [01:35<07:27, 1.04s/it]
17%|#6 | 86/514 [01:36<08:03, 1.13s/it]
17%|#6 | 87/514 [01:37<07:54, 1.11s/it]
17%|#7 | 88/514 [01:38<07:41, 1.08s/it]
17%|#7 | 89/514 [01:40<07:59, 1.13s/it]
18%|#7 | 90/514 [01:41<08:11, 1.16s/it]
18%|#7 | 91/514 [01:42<07:58, 1.13s/it]
18%|#7 | 92/514 [01:43<07:48, 1.11s/it]
18%|#8 | 93/514 [01:44<07:46, 1.11s/it]
18%|#8 | 94/514 [01:45<07:33, 1.08s/it]
18%|#8 | 95/514 [01:46<07:26, 1.06s/it]
19%|#8 | 96/514 [01:47<07:20, 1.05s/it]
19%|#8 | 97/514 [01:48<07:19, 1.05s/it]
19%|#9 | 98/514 [01:49<07:13, 1.04s/it]
19%|#9 | 99/514 [01:50<07:09, 1.03s/it]
19%|#9 | 100/514 [01:51<07:08, 1.04s/it]
20%|#9 | 101/514 [01:52<06:59, 1.02s/it]
20%|#9 | 102/514 [01:53<07:07, 1.04s/it]
20%|## | 103/514 [01:54<07:16, 1.06s/it]
20%|## | 104/514 [01:56<07:18, 1.07s/it]
20%|## | 105/514 [01:57<07:11, 1.05s/it]
21%|## | 106/514 [01:58<07:03, 1.04s/it]
21%|## | 107/514 [01:59<06:58, 1.03s/it]
21%|##1 | 108/514 [02:00<07:05, 1.05s/it]
21%|##1 | 109/514 [02:01<07:08, 1.06s/it]
21%|##1 | 110/514 [02:02<06:56, 1.03s/it]
22%|##1 | 111/514 [02:03<06:56, 1.03s/it]
22%|##1 | 112/514 [02:04<06:57, 1.04s/it]
22%|##1 | 113/514 [02:05<06:59, 1.05s/it]
22%|##2 | 114/514 [02:06<07:22, 1.11s/it]
22%|##2 | 115/514 [02:08<08:48, 1.33s/it]
23%|##2 | 116/514 [02:09<08:42, 1.31s/it]
23%|##2 | 117/514 [02:10<08:14, 1.24s/it]
23%|##2 | 118/514 [02:11<08:01, 1.22s/it]
23%|##3 | 119/514 [02:13<07:56, 1.21s/it]
23%|##3 | 120/514 [02:14<07:24, 1.13s/it]
24%|##3 | 121/514 [02:15<07:33, 1.15s/it]
24%|##3 | 122/514 [02:16<07:16, 1.11s/it]
24%|##3 | 123/514 [02:17<07:29, 1.15s/it]
24%|##4 | 124/514 [02:18<07:13, 1.11s/it]
24%|##4 | 125/514 [02:19<07:01, 1.08s/it]
25%|##4 | 126/514 [02:21<07:44, 1.20s/it]
25%|##4 | 127/514 [02:22<07:54, 1.23s/it]
25%|##4 | 128/514 [02:23<07:26, 1.16s/it]
25%|##5 | 129/514 [02:24<07:04, 1.10s/it]
25%|##5 | 130/514 [02:25<06:53, 1.08s/it]
25%|##5 | 131/514 [02:26<06:49, 1.07s/it]
26%|##5 | 132/514 [02:27<06:42, 1.05s/it]
26%|##5 | 133/514 [02:28<06:41, 1.05s/it]
26%|##6 | 134/514 [02:29<07:05, 1.12s/it]
26%|##6 | 135/514 [02:30<06:52, 1.09s/it]
26%|##6 | 136/514 [02:31<07:06, 1.13s/it]
27%|##6 | 137/514 [02:33<07:18, 1.16s/it]
27%|##6 | 138/514 [02:34<08:07, 1.30s/it]
27%|##7 | 139/514 [02:36<08:06, 1.30s/it]
27%|##7 | 140/514 [02:37<07:23, 1.18s/it]
27%|##7 | 141/514 [02:38<07:01, 1.13s/it]
28%|##7 | 142/514 [02:39<06:53, 1.11s/it]
28%|##7 | 143/514 [02:40<06:36, 1.07s/it]
28%|##8 | 144/514 [02:41<07:02, 1.14s/it]
28%|##8 | 145/514 [02:42<07:10, 1.17s/it]
28%|##8 | 146/514 [02:43<06:55, 1.13s/it]
29%|##8 | 147/514 [02:44<06:53, 1.13s/it]
29%|##8 | 148/514 [02:45<06:47, 1.11s/it]
29%|##8 | 149/514 [02:47<07:43, 1.27s/it]
29%|##9 | 150/514 [02:48<07:46, 1.28s/it]
29%|##9 | 151/514 [02:50<07:48, 1.29s/it]
30%|##9 | 152/514 [02:51<08:09, 1.35s/it]
30%|##9 | 153/514 [02:52<07:28, 1.24s/it]
30%|##9 | 154/514 [02:53<06:59, 1.16s/it]
30%|### | 155/514 [02:54<06:45, 1.13s/it]
30%|### | 156/514 [02:55<06:40, 1.12s/it]
31%|### | 157/514 [02:57<07:50, 1.32s/it]
31%|### | 158/514 [02:58<07:22, 1.24s/it]
31%|### | 159/514 [02:59<07:19, 1.24s/it]
31%|###1 | 160/514 [03:00<07:01, 1.19s/it]
31%|###1 | 161/514 [03:01<06:46, 1.15s/it]
32%|###1 | 162/514 [03:03<06:41, 1.14s/it]
32%|###1 | 163/514 [03:04<06:58, 1.19s/it]
32%|###1 | 164/514 [03:05<06:47, 1.16s/it]
32%|###2 | 165/514 [03:06<06:35, 1.13s/it]
32%|###2 | 166/514 [03:07<06:27, 1.11s/it]
32%|###2 | 167/514 [03:08<06:16, 1.09s/it]
33%|###2 | 168/514 [03:09<06:11, 1.07s/it]
33%|###2 | 169/514 [03:11<07:29, 1.30s/it]
33%|###3 | 170/514 [03:12<07:02, 1.23s/it]
33%|###3 | 171/514 [03:13<06:59, 1.22s/it]
33%|###3 | 172/514 [03:15<07:21, 1.29s/it]
34%|###3 | 173/514 [03:16<06:57, 1.23s/it]
34%|###3 | 174/514 [03:17<06:39, 1.17s/it]
34%|###4 | 175/514 [03:18<06:54, 1.22s/it]
34%|###4 | 176/514 [03:19<06:45, 1.20s/it]
34%|###4 | 177/514 [03:20<06:34, 1.17s/it]
35%|###4 | 178/514 [03:22<06:25, 1.15s/it]
35%|###4 | 179/514 [03:23<06:28, 1.16s/it]
35%|###5 | 180/514 [03:24<06:43, 1.21s/it]
35%|###5 | 181/514 [03:25<06:39, 1.20s/it]
35%|###5 | 182/514 [03:27<07:49, 1.41s/it]
36%|###5 | 183/514 [03:28<07:14, 1.31s/it]
36%|###5 | 184/514 [03:30<07:27, 1.36s/it]
36%|###5 | 185/514 [03:31<07:25, 1.35s/it]
36%|###6 | 186/514 [03:32<07:02, 1.29s/it]
36%|###6 | 187/514 [03:33<07:03, 1.29s/it]
37%|###6 | 188/514 [03:35<07:12, 1.33s/it]
37%|###6 | 189/514 [03:37<07:46, 1.43s/it]
37%|###6 | 190/514 [03:38<07:31, 1.39s/it]
37%|###7 | 191/514 [03:39<07:25, 1.38s/it]
37%|###7 | 192/514 [03:40<06:59, 1.30s/it]
38%|###7 | 193/514 [03:42<06:45, 1.26s/it]
38%|###7 | 194/514 [03:43<06:28, 1.21s/it]
38%|###7 | 195/514 [03:44<06:20, 1.19s/it]
38%|###8 | 196/514 [03:45<06:29, 1.23s/it]
38%|###8 | 197/514 [03:46<06:22, 1.21s/it]
39%|###8 | 198/514 [03:47<06:24, 1.22s/it]
39%|###8 | 199/514 [03:49<06:14, 1.19s/it]
39%|###8 | 200/514 [03:50<06:18, 1.20s/it]
39%|###9 | 201/514 [03:51<06:06, 1.17s/it]
39%|###9 | 202/514 [03:52<06:30, 1.25s/it]
39%|###9 | 203/514 [03:53<06:11, 1.19s/it]
40%|###9 | 204/514 [03:55<06:00, 1.16s/it]
40%|###9 | 205/514 [03:56<06:18, 1.22s/it]
40%|#### | 206/514 [03:57<06:03, 1.18s/it]
40%|#### | 207/514 [03:58<06:14, 1.22s/it]
40%|#### | 208/514 [03:59<06:13, 1.22s/it]
41%|#### | 209/514 [04:01<06:09, 1.21s/it]
41%|#### | 210/514 [04:02<05:52, 1.16s/it]
41%|####1 | 211/514 [04:03<05:36, 1.11s/it]
41%|####1 | 212/514 [04:04<05:54, 1.17s/it]
41%|####1 | 213/514 [04:05<06:15, 1.25s/it]
42%|####1 | 214/514 [04:07<05:59, 1.20s/it]
42%|####1 | 215/514 [04:08<06:42, 1.35s/it]
42%|####2 | 216/514 [04:09<06:09, 1.24s/it]
42%|####2 | 217/514 [04:10<05:42, 1.15s/it]
42%|####2 | 218/514 [04:11<05:54, 1.20s/it]
43%|####2 | 219/514 [04:13<06:15, 1.27s/it]
43%|####2 | 220/514 [04:14<06:28, 1.32s/it]
43%|####2 | 221/514 [04:16<06:29, 1.33s/it]
43%|####3 | 222/514 [04:17<06:07, 1.26s/it]
43%|####3 | 223/514 [04:18<05:49, 1.20s/it]
44%|####3 | 224/514 [04:19<05:32, 1.15s/it]
44%|####3 | 225/514 [04:20<05:21, 1.11s/it]
44%|####3 | 226/514 [04:21<05:15, 1.10s/it]
44%|####4 | 227/514 [04:22<05:51, 1.22s/it]
44%|####4 | 228/514 [04:24<05:42, 1.20s/it]
45%|####4 | 229/514 [04:25<06:11, 1.30s/it]
45%|####4 | 230/514 [04:26<05:51, 1.24s/it]
45%|####4 | 231/514 [04:27<05:36, 1.19s/it]
45%|####5 | 232/514 [04:29<06:10, 1.31s/it]
45%|####5 | 233/514 [04:30<06:07, 1.31s/it]
46%|####5 | 234/514 [04:31<05:54, 1.26s/it]
46%|####5 | 235/514 [04:33<05:45, 1.24s/it]
46%|####5 | 236/514 [04:34<05:27, 1.18s/it]
46%|####6 | 237/514 [04:35<05:26, 1.18s/it]
46%|####6 | 238/514 [04:36<05:44, 1.25s/it]
46%|####6 | 239/514 [04:37<05:29, 1.20s/it]
47%|####6 | 240/514 [04:39<05:51, 1.28s/it]
47%|####6 | 241/514 [04:40<05:32, 1.22s/it]
47%|####7 | 242/514 [04:41<05:53, 1.30s/it]
47%|####7 | 243/514 [04:43<05:49, 1.29s/it]
47%|####7 | 244/514 [04:45<07:18, 1.62s/it]
48%|####7 | 245/514 [04:46<06:47, 1.52s/it]
48%|####7 | 246/514 [04:47<06:09, 1.38s/it]
48%|####8 | 247/514 [04:48<05:35, 1.26s/it]
48%|####8 | 248/514 [04:49<05:30, 1.24s/it]
48%|####8 | 249/514 [04:51<05:34, 1.26s/it]
49%|####8 | 250/514 [04:52<05:34, 1.27s/it]
49%|####8 | 251/514 [04:53<05:22, 1.23s/it]
49%|####9 | 252/514 [04:54<05:19, 1.22s/it]
49%|####9 | 253/514 [04:56<05:41, 1.31s/it]
49%|####9 | 254/514 [04:57<05:46, 1.33s/it]
50%|####9 | 255/514 [04:59<06:04, 1.41s/it]
50%|####9 | 256/514 [05:00<05:36, 1.31s/it]
50%|##### | 257/514 [05:01<05:23, 1.26s/it]
50%|##### | 258/514 [05:03<05:33, 1.30s/it]
50%|##### | 259/514 [05:04<05:24, 1.27s/it]
51%|##### | 260/514 [05:05<05:06, 1.21s/it]
51%|##### | 261/514 [05:06<04:56, 1.17s/it]
51%|##### | 262/514 [05:07<05:12, 1.24s/it]
51%|#####1 | 263/514 [05:09<05:16, 1.26s/it]
51%|#####1 | 264/514 [05:10<05:33, 1.33s/it]
52%|#####1 | 265/514 [05:11<05:24, 1.30s/it]
52%|#####1 | 266/514 [05:13<05:19, 1.29s/it]
52%|#####1 | 267/514 [05:14<05:02, 1.23s/it]
52%|#####2 | 268/514 [05:15<05:01, 1.23s/it]
52%|#####2 | 269/514 [05:16<04:59, 1.22s/it]
53%|#####2 | 270/514 [05:18<05:29, 1.35s/it]
53%|#####2 | 271/514 [05:19<05:42, 1.41s/it]
53%|#####2 | 272/514 [05:21<05:29, 1.36s/it]
53%|#####3 | 273/514 [05:22<05:06, 1.27s/it]
53%|#####3 | 274/514 [05:23<05:07, 1.28s/it]
54%|#####3 | 275/514 [05:24<04:52, 1.23s/it]
54%|#####3 | 276/514 [05:25<04:43, 1.19s/it]
54%|#####3 | 277/514 [05:26<04:44, 1.20s/it]
54%|#####4 | 278/514 [05:27<04:34, 1.16s/it]
54%|#####4 | 279/514 [05:29<04:34, 1.17s/it]
54%|#####4 | 280/514 [05:30<04:25, 1.13s/it]
55%|#####4 | 281/514 [05:31<04:43, 1.22s/it]
55%|#####4 | 282/514 [05:32<04:40, 1.21s/it]
55%|#####5 | 283/514 [05:33<04:31, 1.18s/it]
55%|#####5 | 284/514 [05:34<04:28, 1.17s/it]
55%|#####5 | 285/514 [05:36<04:45, 1.25s/it]
56%|#####5 | 286/514 [05:37<04:32, 1.20s/it]
56%|#####5 | 287/514 [05:39<04:55, 1.30s/it]
56%|#####6 | 288/514 [05:40<05:04, 1.35s/it]
56%|#####6 | 289/514 [05:41<04:48, 1.28s/it]
56%|#####6 | 290/514 [05:42<04:34, 1.22s/it]
57%|#####6 | 291/514 [05:44<04:38, 1.25s/it]
57%|#####6 | 292/514 [05:45<04:53, 1.32s/it]
57%|#####7 | 293/514 [05:46<04:49, 1.31s/it]
57%|#####7 | 294/514 [05:48<04:46, 1.30s/it]
57%|#####7 | 295/514 [05:49<04:46, 1.31s/it]
58%|#####7 | 296/514 [05:50<04:41, 1.29s/it]
58%|#####7 | 297/514 [05:51<04:28, 1.24s/it]
58%|#####7 | 298/514 [05:52<04:10, 1.16s/it]
58%|#####8 | 299/514 [05:54<04:28, 1.25s/it]
58%|#####8 | 300/514 [05:55<04:18, 1.21s/it]
59%|#####8 | 301/514 [05:56<04:06, 1.16s/it]
59%|#####8 | 302/514 [05:57<04:09, 1.18s/it]
59%|#####8 | 303/514 [05:59<04:49, 1.37s/it]
59%|#####9 | 304/514 [06:00<04:39, 1.33s/it]
59%|#####9 | 305/514 [06:01<04:30, 1.30s/it]
60%|#####9 | 306/514 [06:03<04:20, 1.25s/it]
60%|#####9 | 307/514 [06:04<04:16, 1.24s/it]
60%|#####9 | 308/514 [06:05<04:13, 1.23s/it]
60%|###### | 309/514 [06:06<04:00, 1.17s/it]
60%|###### | 310/514 [06:07<03:53, 1.14s/it]
61%|###### | 311/514 [06:08<04:11, 1.24s/it]
61%|###### | 312/514 [06:10<04:08, 1.23s/it]
61%|###### | 313/514 [06:11<04:00, 1.20s/it]
61%|######1 | 314/514 [06:12<03:57, 1.19s/it]
61%|######1 | 315/514 [06:13<03:51, 1.16s/it]
61%|######1 | 316/514 [06:14<03:52, 1.17s/it]
62%|######1 | 317/514 [06:15<03:47, 1.16s/it]
62%|######1 | 318/514 [06:17<03:42, 1.14s/it]
62%|######2 | 319/514 [06:18<03:43, 1.14s/it]
62%|######2 | 320/514 [06:20<05:17, 1.64s/it]
62%|######2 | 321/514 [06:22<04:42, 1.47s/it]
63%|######2 | 322/514 [06:23<04:18, 1.34s/it]
63%|######2 | 323/514 [06:24<04:06, 1.29s/it]
63%|######3 | 324/514 [06:25<03:56, 1.25s/it]
63%|######3 | 325/514 [06:26<03:55, 1.25s/it]
63%|######3 | 326/514 [06:27<03:57, 1.26s/it]
64%|######3 | 327/514 [06:29<04:07, 1.32s/it]
64%|######3 | 328/514 [06:30<04:20, 1.40s/it]
64%|######4 | 329/514 [06:32<04:25, 1.44s/it]
64%|######4 | 330/514 [06:33<04:04, 1.33s/it]
64%|######4 | 331/514 [06:34<03:51, 1.26s/it]
65%|######4 | 332/514 [06:35<03:41, 1.22s/it]
65%|######4 | 333/514 [06:36<03:36, 1.20s/it]
65%|######4 | 334/514 [06:38<03:48, 1.27s/it]
65%|######5 | 335/514 [06:39<03:52, 1.30s/it]
65%|######5 | 336/514 [06:41<03:55, 1.32s/it]
66%|######5 | 337/514 [06:42<03:57, 1.34s/it]
66%|######5 | 338/514 [06:43<03:50, 1.31s/it]
66%|######5 | 339/514 [06:45<03:46, 1.29s/it]
66%|######6 | 340/514 [06:46<03:50, 1.33s/it]
66%|######6 | 341/514 [06:47<03:42, 1.28s/it]
67%|######6 | 342/514 [06:48<03:35, 1.25s/it]
67%|######6 | 343/514 [06:50<04:02, 1.42s/it]
67%|######6 | 344/514 [06:51<03:51, 1.36s/it]
67%|######7 | 345/514 [06:53<04:09, 1.48s/it]
67%|######7 | 346/514 [06:54<03:50, 1.37s/it]
68%|######7 | 347/514 [06:55<03:43, 1.34s/it]
68%|######7 | 348/514 [06:57<03:36, 1.30s/it]
68%|######7 | 349/514 [06:58<03:38, 1.33s/it]
68%|######8 | 350/514 [07:00<03:56, 1.44s/it]
68%|######8 | 351/514 [07:02<04:20, 1.60s/it]
68%|######8 | 352/514 [07:04<05:12, 1.93s/it]
69%|######8 | 353/514 [07:07<05:31, 2.06s/it]
69%|######8 | 354/514 [07:09<05:37, 2.11s/it]
69%|######9 | 355/514 [07:11<05:20, 2.02s/it]
69%|######9 | 356/514 [07:13<05:03, 1.92s/it]
69%|######9 | 357/514 [07:14<04:55, 1.88s/it]
70%|######9 | 358/514 [07:16<04:53, 1.88s/it]
70%|######9 | 359/514 [07:18<04:51, 1.88s/it]
70%|####### | 360/514 [07:20<04:43, 1.84s/it]
70%|####### | 361/514 [07:21<04:25, 1.73s/it]
70%|####### | 362/514 [07:23<04:31, 1.79s/it]
71%|####### | 363/514 [07:26<05:02, 2.01s/it]
71%|####### | 364/514 [07:27<04:43, 1.89s/it]
71%|#######1 | 365/514 [07:29<04:42, 1.90s/it]
71%|#######1 | 366/514 [07:31<04:34, 1.86s/it]
71%|#######1 | 367/514 [07:33<04:24, 1.80s/it]
72%|#######1 | 368/514 [07:35<04:44, 1.95s/it]
72%|#######1 | 369/514 [07:37<04:35, 1.90s/it]
72%|#######1 | 370/514 [07:38<04:13, 1.76s/it]
72%|#######2 | 371/514 [07:40<04:02, 1.69s/it]
72%|#######2 | 372/514 [07:42<04:14, 1.79s/it]
73%|#######2 | 373/514 [07:43<04:07, 1.76s/it]
73%|#######2 | 374/514 [07:45<04:04, 1.75s/it]
73%|#######2 | 375/514 [07:47<04:07, 1.78s/it]
73%|#######3 | 376/514 [07:49<04:00, 1.75s/it]
73%|#######3 | 377/514 [07:50<03:54, 1.71s/it]
74%|#######3 | 378/514 [07:52<04:02, 1.78s/it]
74%|#######3 | 379/514 [07:54<03:58, 1.77s/it]
74%|#######3 | 380/514 [07:56<03:56, 1.77s/it]
74%|#######4 | 381/514 [07:58<04:07, 1.86s/it]
74%|#######4 | 382/514 [08:00<04:00, 1.82s/it]
75%|#######4 | 383/514 [08:01<03:50, 1.76s/it]
75%|#######4 | 384/514 [08:03<03:43, 1.72s/it]
75%|#######4 | 385/514 [08:04<03:37, 1.68s/it]
75%|#######5 | 386/514 [08:06<03:47, 1.78s/it]
75%|#######5 | 387/514 [08:08<03:43, 1.76s/it]
75%|#######5 | 388/514 [08:10<03:45, 1.79s/it]
76%|#######5 | 389/514 [08:12<03:36, 1.73s/it]
76%|#######5 | 390/514 [08:13<03:32, 1.71s/it]
76%|#######6 | 391/514 [08:15<03:40, 1.79s/it]
76%|#######6 | 392/514 [08:17<03:43, 1.83s/it]
76%|#######6 | 393/514 [08:20<04:07, 2.05s/it]
77%|#######6 | 394/514 [08:22<04:03, 2.03s/it]
77%|#######6 | 395/514 [08:23<03:44, 1.89s/it]
77%|#######7 | 396/514 [08:25<03:52, 1.97s/it]
77%|#######7 | 397/514 [08:27<03:40, 1.89s/it]
77%|#######7 | 398/514 [08:29<03:31, 1.82s/it]
78%|#######7 | 399/514 [08:31<03:37, 1.89s/it]
78%|#######7 | 400/514 [08:33<03:29, 1.84s/it]
78%|#######8 | 401/514 [08:34<03:20, 1.77s/it]
78%|#######8 | 402/514 [08:36<03:17, 1.76s/it]
78%|#######8 | 403/514 [08:38<03:13, 1.74s/it]
79%|#######8 | 404/514 [08:39<03:09, 1.72s/it]
79%|#######8 | 405/514 [08:41<03:07, 1.72s/it]
79%|#######8 | 406/514 [08:43<03:09, 1.75s/it]
79%|#######9 | 407/514 [08:45<03:05, 1.73s/it]
79%|#######9 | 408/514 [08:47<03:12, 1.82s/it]
80%|#######9 | 409/514 [08:48<03:04, 1.76s/it]
80%|#######9 | 410/514 [08:50<03:03, 1.76s/it]
80%|#######9 | 411/514 [08:52<02:56, 1.71s/it]
80%|######## | 412/514 [08:53<02:49, 1.66s/it]
80%|######## | 413/514 [08:55<02:46, 1.65s/it]
81%|######## | 414/514 [08:57<02:52, 1.72s/it]
81%|######## | 415/514 [08:58<02:45, 1.67s/it]
81%|######## | 416/514 [09:00<02:52, 1.76s/it]
81%|########1 | 417/514 [09:02<02:47, 1.73s/it]
81%|########1 | 418/514 [09:03<02:44, 1.72s/it]
82%|########1 | 419/514 [09:05<02:39, 1.68s/it]
82%|########1 | 420/514 [09:07<02:32, 1.62s/it]
82%|########1 | 421/514 [09:08<02:25, 1.57s/it]
82%|########2 | 422/514 [09:10<02:24, 1.57s/it]
82%|########2 | 423/514 [09:11<02:21, 1.55s/it]
82%|########2 | 424/514 [09:13<02:29, 1.66s/it]
83%|########2 | 425/514 [09:14<02:24, 1.62s/it]
83%|########2 | 426/514 [09:16<02:21, 1.61s/it]
83%|########3 | 427/514 [09:18<02:19, 1.60s/it]
83%|########3 | 428/514 [09:19<02:16, 1.58s/it]
83%|########3 | 429/514 [09:21<02:16, 1.61s/it]
84%|########3 | 430/514 [09:22<02:15, 1.61s/it]
84%|########3 | 431/514 [09:24<02:18, 1.67s/it]
84%|########4 | 432/514 [09:26<02:16, 1.67s/it]
84%|########4 | 433/514 [09:27<02:11, 1.63s/it]
84%|########4 | 434/514 [09:29<02:06, 1.58s/it]
85%|########4 | 435/514 [09:31<02:05, 1.59s/it]
85%|########4 | 436/514 [09:33<02:24, 1.85s/it]
85%|########5 | 437/514 [09:35<02:17, 1.79s/it]
85%|########5 | 438/514 [09:37<02:29, 1.97s/it]
85%|########5 | 439/514 [09:39<02:29, 2.00s/it]
86%|########5 | 440/514 [09:41<02:26, 1.99s/it]
86%|########5 | 441/514 [09:43<02:23, 1.97s/it]
86%|########5 | 442/514 [09:45<02:18, 1.92s/it]
86%|########6 | 443/514 [09:47<02:12, 1.86s/it]
86%|########6 | 444/514 [09:49<02:26, 2.09s/it]
87%|########6 | 445/514 [09:51<02:19, 2.02s/it]
87%|########6 | 446/514 [09:53<02:21, 2.08s/it]
87%|########6 | 447/514 [09:55<02:15, 2.03s/it]
87%|########7 | 448/514 [09:58<02:20, 2.12s/it]
87%|########7 | 449/514 [09:59<02:07, 1.95s/it]
88%|########7 | 450/514 [10:01<02:06, 1.98s/it]
88%|########7 | 451/514 [10:03<02:03, 1.95s/it]
88%|########7 | 452/514 [10:05<01:57, 1.90s/it]
88%|########8 | 453/514 [10:07<01:58, 1.95s/it]
88%|########8 | 454/514 [10:08<01:51, 1.86s/it]
89%|########8 | 455/514 [10:10<01:50, 1.87s/it]
89%|########8 | 456/514 [10:12<01:50, 1.91s/it]
89%|########8 | 457/514 [10:14<01:43, 1.81s/it]
89%|########9 | 458/514 [10:16<01:43, 1.84s/it]
89%|########9 | 459/514 [10:18<01:43, 1.88s/it]
89%|########9 | 460/514 [10:20<01:45, 1.95s/it]
90%|########9 | 461/514 [10:22<01:48, 2.05s/it]
90%|########9 | 462/514 [10:24<01:37, 1.88s/it]
90%|######### | 463/514 [10:25<01:30, 1.77s/it]
90%|######### | 464/514 [10:27<01:25, 1.71s/it]
90%|######### | 465/514 [10:29<01:29, 1.83s/it]
91%|######### | 466/514 [10:31<01:27, 1.83s/it]
91%|######### | 467/514 [10:33<01:25, 1.81s/it]
91%|#########1| 468/514 [10:34<01:22, 1.79s/it]
91%|#########1| 469/514 [10:36<01:18, 1.74s/it]
91%|#########1| 470/514 [10:37<01:13, 1.67s/it]
92%|#########1| 471/514 [10:41<01:30, 2.11s/it]
92%|#########1| 472/514 [10:43<01:30, 2.15s/it]
92%|#########2| 473/514 [10:45<01:26, 2.10s/it]
92%|#########2| 474/514 [10:47<01:23, 2.10s/it]
92%|#########2| 475/514 [10:49<01:18, 2.01s/it]
93%|#########2| 476/514 [10:50<01:11, 1.89s/it]
93%|#########2| 477/514 [10:53<01:19, 2.14s/it]
93%|#########2| 478/514 [10:55<01:11, 1.99s/it]
93%|#########3| 479/514 [10:56<01:07, 1.94s/it]
93%|#########3| 480/514 [11:01<01:31, 2.69s/it]
94%|#########3| 481/514 [11:03<01:22, 2.51s/it]
94%|#########3| 482/514 [11:05<01:13, 2.31s/it]
94%|#########3| 483/514 [11:07<01:07, 2.17s/it]
94%|#########4| 484/514 [11:08<00:59, 1.98s/it]
94%|#########4| 485/514 [11:10<00:55, 1.93s/it]
95%|#########4| 486/514 [11:12<00:54, 1.94s/it]
95%|#########4| 487/514 [11:14<00:51, 1.92s/it]
95%|#########4| 488/514 [11:15<00:47, 1.83s/it]
95%|#########5| 489/514 [11:17<00:43, 1.74s/it]
95%|#########5| 490/514 [11:19<00:41, 1.75s/it]
96%|#########5| 491/514 [11:21<00:41, 1.79s/it]
96%|#########5| 492/514 [11:23<00:39, 1.81s/it]
96%|#########5| 493/514 [11:24<00:38, 1.82s/it]
96%|#########6| 494/514 [11:27<00:38, 1.92s/it]
96%|#########6| 495/514 [11:28<00:35, 1.86s/it]
96%|#########6| 496/514 [11:30<00:32, 1.82s/it]
97%|#########6| 497/514 [11:32<00:31, 1.83s/it]
97%|#########6| 498/514 [11:34<00:28, 1.80s/it]
97%|#########7| 499/514 [11:35<00:26, 1.80s/it]
97%|#########7| 500/514 [11:37<00:24, 1.71s/it]
97%|#########7| 501/514 [11:39<00:21, 1.69s/it]
98%|#########7| 502/514 [11:40<00:20, 1.67s/it]
98%|#########7| 503/514 [11:42<00:20, 1.83s/it]
98%|#########8| 504/514 [11:44<00:17, 1.78s/it]
98%|#########8| 505/514 [11:46<00:16, 1.79s/it]
98%|#########8| 506/514 [11:48<00:14, 1.77s/it]
99%|#########8| 507/514 [11:49<00:12, 1.80s/it]
99%|#########8| 508/514 [11:51<00:10, 1.81s/it]
99%|#########9| 509/514 [11:53<00:08, 1.70s/it]
99%|#########9| 510/514 [11:54<00:06, 1.67s/it]
99%|#########9| 511/514 [11:56<00:05, 1.71s/it]
100%|#########9| 512/514 [11:58<00:03, 1.79s/it]
100%|#########9| 513/514 [12:00<00:01, 1.92s/it]
100%|##########| 514/514 [12:02<00:00, 1.88s/it]
100%|##########| 514/514 [12:02<00:00, 1.41s/it]