STUNT: Predictions, Additional Analysis and Results

We experimented with the following configurations for the Hyper Parameter Tuning phase of STUNT :

Learning Rate	Parameters	Pre-Trained	No Pre-Trained
C-LR	LR	0.001	0.001
ST-LR	LRstarting LRmax Ratio Cut	0.001 0.01 32 0.1	0.001 0.01 32 0.1
ISQ-LR	LRstarting Warmup	0.01 10.000	0.01 10.000
PD-LR	LRstarting LRend Power	0.01 0.01 0.5	0.001 0.001 0.5

The results achieved by the four configuations on the evaluation set, for the summarization task, are reported in the following table:

Summarization

The following table reports the average metrics scores achieved by STUNT:

Learning Rate	BLEU-4	METEOR	ROUGE Precision	Rouge Recall
C-LR	0.45	0.44	0.35	0.39
ST-LR	0.41	0.40	0.33	0.37
ISQ-LR	0.46	0.44	0.35	0.40
PD-LR	0.45	0.43	0.35	0.39

We released all our trained models. You can find:

1. the trained STUNT model here
2. the STUNT model trained on the dataset released in Towards automatically generating block comments for code snippets here
3. the pretrained model used as backbone for STUNT here

The predictions generated by STUNT and the Jaccard baseline can be found here, together with the metrics in terms o BLEU score, METEOR, and ROUGE.
The results achieved by running STUNT on the dataset used to train RL-ComBlock can be found here.
The scripts to compute the BLEU score and the Quantitative Metrics can be found here here.