Yoochoose

This dataset contains user sessions from an unnamed e-commerce site. The data is already split into proper sessions and each session consists of one or more click events and might have purchase event(s) associated with it. As we are focusing on the next item prediction task within sessions, we only use the click events. The dataset was originally released for RecSys Challenge 2015 and has been used for evaluating sessionbased recommenders since then.

Recall@20

BPR-Max

Cross-Entropy

MRR@20

BPR-Max

Cross-Entropy

GRU4REC-pytorch

Metric

BPR-Max

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.1745	0.1745	0.4346	0.2675	0.5664	0.2851	0.6799	0.2931
GRU4Rec Official	GRU4REC-pytorch params	0.0988	0.0988	0.2613	0.1554	0.3620	0.1688	0.4655	0.1760
GRU4REC-pytorch	OOB	0.0002	0.0002	0.0012	0.0005	0.0031	0.0007	0.0087	0.0011
GRU4REC-pytorch	OOB Correct Eval	0.0009	0.0009	0.0104	0.0036	0.0409	0.0074	0.1066	0.0118
GRU4REC-pytorch	Correct full	0.0108	0.0108	0.0603	0.0269	0.1112	0.0335	0.1923	0.0391

Cross-Entropy

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.1797	0.1797	0.4457	0.2757	0.5698	0.2924	0.6804	0.3002
GRU4Rec Official	GRU4REC-pytorch params	0.0717	0.0717	0.2386	0.1301	0.3478	0.1446	0.4583	0.1523
GRU4REC-pytorch	OOB	0.0933	0.0933	0.1090	0.0998	0.1129	0.1003	0.1169	0.1006
GRU4REC-pytorch	OOB Correct Eval	0.0951	0.0951	0.1134	0.1029	0.1173	0.1034	0.1212	0.1037
GRU4REC-pytorch	Correct full	0.0493	0.0493	0.1997	0.1001	0.3052	0.1141	0.4271	0.1227

Metric Diff

BPR-Max

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	GRU4REC-pytorch params	-43.42%	-43.42%	-39.88%	-41.92%	-36.09%	-40.80%	-31.53%	-39.96%
GRU4REC-pytorch	OOB	-99.91%	-99.91%	-99.72%	-99.83%	-99.46%	-99.76%	-98.72%	-99.63%
GRU4REC-pytorch	OOB Correct Eval	-99.48%	-99.48%	-97.62%	-98.66%	-92.78%	-97.40%	-84.32%	-95.98%
GRU4REC-pytorch	Correct full	-93.80%	-93.80%	-86.13%	-89.95%	-80.36%	-88.25%	-71.72%	-86.67%

Cross-Entropy

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	GRU4REC-pytorch params	-60.10%	-60.10%	-46.48%	-52.83%	-38.96%	-50.54%	-32.64%	-49.27%
GRU4REC-pytorch	OOB	-48.10%	-48.10%	-75.54%	-63.82%	-80.19%	-65.70%	-82.82%	-66.50%
GRU4REC-pytorch	OOB Correct Eval	-47.07%	-47.07%	-74.56%	-62.69%	-79.42%	-64.64%	-82.19%	-65.46%
GRU4REC-pytorch	Correct full	-72.58%	-72.58%	-55.19%	-63.68%	-46.44%	-60.98%	-37.22%	-59.14%

Hyperparameters

BPR-Max

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	GRU4REC-pytorch	GRU4REC-pytorch	GRU4REC-pytorch
Variant	Best params	GRU4REC-pytorch params	OOB	OOB Correct Eval	Correct full
loss	bpr-max	bpr-max	bpr-max	bpr-max	bpr-max
optim	adagrad	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False	False
embedding	0	448	448	448	448
final_act	linear	linear	linear	linear	linear
layers	448	448	448	448	448
batch_size	48	48	48	48	48
dropout_p_embed	0.25	0.25	N/A	N/A	0.25
dropout_p_hidden	0	0	N/A	N/A	0
learning_rate	0.075	0.075	0.075	0.075	0.075
momentum	0.1	0	N/A	N/A	N/A
n_sample	2048	0	N/A	N/A	N/A
sample_alpha	0.2	0	N/A	N/A	N/A
bpreg	0.5	0	N/A	N/A	N/A
logq	0	0	N/A	N/A	N/A

Cross-Entropy

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	GRU4REC-pytorch	GRU4REC-pytorch	GRU4REC-pytorch
Variant	Best params	GRU4REC-pytorch params	OOB	OOB Correct Eval	Correct full
loss	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy
optim	adagrad	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False	False
embedding	0	480	480	480	480
final_act	softmax	softmax	softmax	softmax	softmax
layers	480	480	480	480	480
batch_size	48	48	48	48	48
dropout_p_embed	0	0	N/A	N/A	0
dropout_p_hidden	0.2	0.2	N/A	N/A	0.2
learning_rate	0.07	0.07	0.07	0.07	0.07
momentum	0	0	N/A	N/A	N/A
n_sample	2048	0	N/A	N/A	N/A
sample_alpha	0.2	0	N/A	N/A	N/A
bpreg	0	0	N/A	N/A	N/A
logq	1	0	N/A	N/A	N/A

Runtimes

BPR-Max

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	487.51			919.23	44121.00
GRU4Rec Official	GRU4REC-pytorch params	362.75	0.74 x		1235.40	59297.00
GRU4REC-pytorch	OOB	1854.78	3.80 x	5.11 x	241.61	11597.27
GRU4REC-pytorch	OOB Correct Eval	1857.10	3.81 x	5.12 x	241.30	11582.50
GRU4REC-pytorch	Correct full	2123.40	4.36 x	5.85 x	211.06	10130.73

Cross-Entropy

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	451.75			991.99	47613.00
GRU4Rec Official	GRU4REC-pytorch params	350.38	0.78 x		1279.01	61390.00
GRU4REC-pytorch	OOB	1948.18	4.31 x	5.56 x	230.02	11040.99
GRU4REC-pytorch	OOB Correct Eval	1944.45	4.30 x	5.55 x	230.46	11062.17
GRU4REC-pytorch	Correct full	1933.25	4.28 x	5.52 x	231.80	11126.37

Torch-GRU4Rec

Metric

BPR-Max

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.1745	0.1745	0.4346	0.2675	0.5664	0.2851	0.6799	0.2931
GRU4Rec Official	Torch-GRU4Rec params	0.1748	0.1748	0.4298	0.2662	0.5603	0.2837	0.6769	0.2919
Torch-GRU4Rec	OOB	0.1755	0.1755	0.4271	0.2654	0.5560	0.2826	0.6711	0.2907

Cross-Entropy

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.1797	0.1797	0.4457	0.2757	0.5698	0.2924	0.6804	0.3002
GRU4Rec Official	Torch-GRU4Rec params	0.1710	0.1710	0.4301	0.2633	0.5571	0.2803	0.6690	0.2882
Torch-GRU4Rec	OOB	0.1686	0.1686	0.4268	0.2598	0.5528	0.2768	0.6671	0.2847

Metric Diff

BPR-Max

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	Torch-GRU4Rec params	0.14%	0.14%	-1.09%	-0.49%	-1.07%	-0.48%	-0.45%	-0.39%
Torch-GRU4Rec	OOB	0.53%	0.53%	-1.72%	-0.77%	-1.84%	-0.89%	-1.30%	-0.82%

Cross-Entropy

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	Torch-GRU4Rec params	-4.81%	-4.81%	-3.50%	-4.52%	-2.23%	-4.14%	-1.67%	-4.00%
Torch-GRU4Rec	OOB	-6.14%	-6.14%	-4.25%	-5.77%	-2.99%	-5.35%	-1.95%	-5.14%

Hyperparameters

BPR-Max

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	Torch-GRU4Rec
Variant	Best params	Torch-GRU4Rec params	OOB
loss	bpr-max	bpr-max	bpr-max
optim	adagrad	adagrad	adagrad
constrained_embedding	True	False	False
embedding	0	448	448
final_act	linear	linear	linear
layers	448	448	448
batch_size	48	48	48
dropout_p_embed	0.25	0.25	0.25
dropout_p_hidden	0	0	0
learning_rate	0.075	0.075	0.075
momentum	0.1	0	N/A
n_sample	2048	2048	2048
sample_alpha	0.2	0.2	0.2
bpreg	0.5	0.5	0.5
logq	0	0	N/A

Cross-Entropy

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	Torch-GRU4Rec
Variant	Best params	Torch-GRU4Rec params	OOB
loss	cross-entropy	cross-entropy	cross-entropy
optim	adagrad	adagrad	adagrad
constrained_embedding	True	False	False
embedding	0	480	480
final_act	softmax	softmax	softmax
layers	480	480	480
batch_size	48	48	48
dropout_p_embed	0	0	0
dropout_p_hidden	0.2	0.2	0.2
learning_rate	0.07	0.07	0.07
momentum	0	0	N/A
n_sample	2048	2048	2048
sample_alpha	0.2	0.2	0.2
bpreg	0	0	0
logq	1	0	N/A

Runtimes

BPR-Max

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	487.51			919.23	44121.00
GRU4Rec Official	Torch-GRU4Rec params	461.22	0.95 x		971.64	46636.00
Torch-GRU4Rec	OOB	2164.21	4.44 x	4.69 x	207.09	9939.82

Cross-Entropy

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	451.75			991.99	47613.00
GRU4Rec Official	Torch-GRU4Rec params	439.19	0.97 x		1020.37	48975.00
Torch-GRU4Rec	OOB	2082.11	4.61 x	4.74 x	215.24	10330.85

Recpack

Metric

BPR-Max

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.1745	0.1745	0.4346	0.2675	0.5664	0.2851	0.6799	0.2931
GRU4Rec Official	Recpack params	0.1753	0.1753	0.4304	0.2666	0.5578	0.2837	0.6749	0.2919
Recpack	OOB	0.1500	0.1500	0.3878	0.2358	0.5182	0.2532	0.6430	0.2619
Recpack	Correct exp	0.1571	0.1571	0.3984	0.2443	0.5260	0.2613	0.6464	0.2697
Recpack	Correct full	0.1581	0.1581	0.3967	0.2437	0.5239	0.2607	0.6433	0.2691

Cross-Entropy

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.1797	0.1797	0.4457	0.2757	0.5698	0.2924	0.6804	0.3002
GRU4Rec Official	Recpack params	0.1709	0.1709	0.4301	0.2625	0.5603	0.2800	0.6713	0.2877
GRU4Rec Official	Recpack params	0.1825	0.1825	0.4447	0.2770	0.5698	0.2938	0.6802	0.3016
Recpack	OOB	0.1533	0.1533	0.3941	0.2393	0.5108	0.2549	0.6214	0.2627
Recpack	Correct exp	0.1548	0.1548	0.3970	0.2417	0.5157	0.2576	0.6256	0.2653
Recpack	Correct full	0.1647	0.1647	0.4074	0.2516	0.5283	0.2678	0.6382	0.2755

Metric Diff

BPR-Max

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	Recpack params	0.43%	0.43%	-0.96%	-0.32%	-1.51%	-0.50%	-0.74%	-0.41%
Recpack	OOB	-14.04%	-14.04%	-10.77%	-11.85%	-8.51%	-11.20%	-5.44%	-10.63%
Recpack	Correct exp	-9.98%	-9.98%	-8.32%	-8.68%	-7.13%	-8.35%	-4.93%	-7.97%
Recpack	Correct full	-9.43%	-9.43%	-8.71%	-8.88%	-7.50%	-8.56%	-5.39%	-8.19%

Cross-Entropy

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	Recpack params	-4.89%	-4.89%	-3.52%	-4.81%	-1.67%	-4.25%	-1.33%	-4.14%
GRU4Rec Official	Recpack params	1.56%	1.56%	-0.23%	0.46%	0.00%	0.50%	-0.02%	0.46%
Recpack	OOB	-14.70%	-14.70%	-11.57%	-13.20%	-10.35%	-12.81%	-8.67%	-12.50%
Recpack	Correct exp	-13.87%	-13.87%	-10.94%	-12.35%	-9.50%	-11.89%	-8.05%	-11.62%
Recpack	Correct full	-8.33%	-8.33%	-8.59%	-8.77%	-7.29%	-8.41%	-6.19%	-8.23%

Hyperparameters

BPR-Max

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	Recpack	Recpack	Recpack
Variant	Best params	Recpack params	OOB	Correct exp	Correct full
loss	bpr-max	bpr-max	bpr-max	bpr-max	bpr-max
optim	adagrad	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False	False
embedding	0	448	448	448	448
final_act	linear	linear	N/A	N/A	N/A
layers	448	448	448	448	448
batch_size	48	48	48	48	48
dropout_p_embed	0.25	0.25	0.25	0.25	0.25
dropout_p_hidden	0	0	0.25	0.25	0
learning_rate	0.075	0.075	0.075	0.075	0.075
momentum	0.1	0	N/A	N/A	N/A
n_sample	2048	2048	2048	2048	2048
sample_alpha	0.2	0	N/A	N/A	N/A
bpreg	0.5	0.5	0.5	0.5	0.5
logq	0	0	N/A	N/A	N/A

Cross-Entropy

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	GRU4Rec Official	Recpack	Recpack	Recpack
Variant	Best params	Recpack params	Recpack params	OOB	Correct exp	Correct full
loss	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy
optim	adagrad	adagrad	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False	False	False
embedding	0	480	480	480	480	480
final_act	softmax	softmax	softmax	softmax	softmax	softmax
layers	480	480	480	480	480	480
batch_size	48	48	48	48	48	48
dropout_p_embed	0	0	0	0	0	0
dropout_p_hidden	0.2	0.2	0.2	0	0	0.2
learning_rate	0.07	0.07	0.07	0.07	0.07	0.07
momentum	0	0	0	N/A	N/A	N/A
n_sample	2048	2048	ALL	N/A	N/A	N/A
sample_alpha	0.2	0	N/A	N/A	N/A	N/A
bpreg	0	0	0	0	0	0
logq	1	0	N/A	N/A	N/A	N/A

Runtimes

BPR-Max

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	487.51			919.23	44121.00
GRU4Rec Official	Recpack params	460.73	0.95 x		972.67	46686.00
Recpack	OOB	21269.54	43.63 x	46.16 x	131.16	1011.29
Recpack	Correct exp	21382.12	43.86 x	46.41 x	130.47	1005.96
Recpack	Correct full	21137.61	43.36 x	45.88 x	131.98	1017.60

Cross-Entropy

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	451.75			991.99	47613.00
GRU4Rec Official	Recpack params	442.76	0.98 x		1012.15	48581.00
GRU4Rec Official	Recpack params	2859.14	6.33 x	6.46 x	156.74	7523.00
Recpack	OOB	13458.18	29.79 x	30.40 x	207.29	1598.25
Recpack	Correct exp	13462.56	29.80 x	30.41 x	207.22	1597.74
Recpack	Correct full	13693.73	30.31 x	30.93 x	203.72	1570.76

GRU4Rec_Tensorflow

Metric

BPR-Max

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Cross-Entropy

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.1797	0.1797	0.4457	0.2757	0.5698	0.2924	0.6804	0.3002
GRU4Rec Official	GRU4Rec_Tensorflow params	0.0715	0.0715	0.2325	0.1269	0.3431	0.1416	0.4555	0.1495
GRU4Rec_Tensorflow	OOB	0.0727	0.0727	0.2051	0.1178	0.2946	0.1296	0.3857	0.1359
GRU4Rec_Tensorflow	Correct Exp	0.0691	0.0691	0.2301	0.1252	0.3409	0.1399	0.4606	0.1482

Metric Diff

BPR-Max

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Cross-Entropy

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	GRU4Rec_Tensorflow params	-60.19%	-60.19%	-47.85%	-53.96%	-39.79%	-51.56%	-33.05%	-50.21%
GRU4Rec_Tensorflow	OOB	-59.52%	-59.52%	-54.00%	-57.27%	-48.30%	-55.68%	-43.31%	-54.73%
GRU4Rec_Tensorflow	Correct Exp	-61.54%	-61.54%	-48.37%	-54.61%	-40.17%	-52.16%	-32.29%	-50.63%

Hyperparameters

BPR-Max

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Cross-Entropy

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	GRU4Rec_Tensorflow	GRU4Rec_Tensorflow
Variant	Best params	GRU4Rec_Tensorflow params	OOB	Correct Exp
loss	cross-entropy	cross-entropy	cross-entropy	cross-entropy
optim	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False
embedding	0	480	480	480
final_act	softmax	softmax	softmax	softmax
layers	480	480	480	480
batch_size	48	48	50	48
dropout_p_embed	0	0	N/A	N/A
dropout_p_hidden	0.2	0.2	0.2	0.2
learning_rate	0.07	0.07	0.07	0.07
momentum	0	0	N/A	N/A
n_sample	2048	0	N/A	N/A
sample_alpha	0.2	0	N/A	N/A
bpreg	0	0	N/A	N/A
logq	1	0	N/A	N/A

Runtimes

BPR-Max

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Cross-Entropy

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	451.75			991.99	47613.00
GRU4Rec Official	GRU4Rec_Tensorflow params	352.48	0.78 x		1271.36	61023.00
GRU4Rec_Tensorflow	OOB	981.87	2.17 x	2.79 x	440.53	22026.42
GRU4Rec_Tensorflow	Correct Exp	965.98	2.14 x	2.74 x	463.92	22268.35

KerasGRU4Rec

Metric

BPR-Max

Note

BPR-Max is not supported by KerasGRU4Rec

Cross-Entropy

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.1797	0.1797	0.4457	0.2757	0.5698	0.2924	0.6804	0.3002
GRU4Rec Official	KerasGRU4Rec params	0.1734	0.1734	0.4351	0.2671	0.5612	0.2841	0.6722	0.2919
GRU4Rec Official	KerasGRU4Rec params	0.1851	0.1851	0.4490	0.2802	0.5724	0.2968	0.6809	0.3044
KerasGRU4Rec	OOB	0.1576	0.1576	0.3915	0.2410	0.5195	0.2581	0.6392	0.2666
KerasGRU4Rec	Correct exp	0.1815	0.1815	0.4444	0.2766	0.5694	0.2934	0.6784	0.3010
KerasGRU4Rec	Correct full	0.1824	0.1824	0.4446	0.2771	0.5678	0.2936	0.6768	0.3013

Metric Diff

BPR-Max

Note

BPR-Max is not supported by KerasGRU4Rec

Cross-Entropy

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	KerasGRU4Rec params	-3.47%	-3.47%	-2.40%	-3.12%	-1.50%	-2.84%	-1.21%	-2.76%
GRU4Rec Official	KerasGRU4Rec params	3.00%	3.00%	0.72%	1.61%	0.45%	1.49%	0.08%	1.39%
KerasGRU4Rec	OOB	-12.28%	-12.28%	-12.17%	-12.58%	-8.84%	-11.73%	-6.04%	-11.20%
KerasGRU4Rec	Correct exp	1.03%	1.03%	-0.29%	0.32%	-0.07%	0.35%	-0.28%	0.29%
KerasGRU4Rec	Correct full	1.52%	1.52%	-0.27%	0.48%	-0.35%	0.42%	-0.52%	0.37%

Hyperparameters

BPR-Max

Note

BPR-Max is not supported by KerasGRU4Rec

Cross-Entropy

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	GRU4Rec Official	KerasGRU4Rec	KerasGRU4Rec	KerasGRU4Rec
Variant	Best params	KerasGRU4Rec params	KerasGRU4Rec params	OOB	Correct exp	Correct full
loss	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy
optim	adagrad	adagrad	adagrad	adam	adagrad	adagrad
constrained_embedding	True	False	False	False	False	False
embedding	0	0	0	0	0	0
final_act	softmax	softmax	softmax	softmax	softmax	softmax
layers	480	480	480	100	480	480
batch_size	48	48	48	48	48	48
dropout_p_embed	0	0	0	N/A	N/A	N/A
dropout_p_hidden	0.2	0.2	0.2	0.25	0.2	0.2
learning_rate	0.07	0.07	0.07	0.001	0.07	0.07
momentum	0	0	0	N/A	N/A	N/A
n_sample	2048	2048	ALL	N/A	N/A	N/A
sample_alpha	0.2	0	N/A	N/A	N/A	N/A
bpreg	0	0	0	N/A	N/A	N/A
logq	1	0	N/A	N/A	N/A	N/A

Runtimes

BPR-Max

Note

BPR-Max is not supported by KerasGRU4Rec

Cross-Entropy

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	451.75			991.99	47613.00
GRU4Rec Official	KerasGRU4Rec params	390.87	0.87 x		1146.51	55030.00
GRU4Rec Official	KerasGRU4Rec params	2800.25	6.20 x	7.16 x	160.03	7681.00
KerasGRU4Rec	OOB	13740.04	30.42 x	35.15 x	32.61	1565.46
KerasGRU4Rec	Correct exp	16610.50	36.77 x	42.50 x	26.98	1294.93
KerasGRU4Rec	Correct full	16666.31	36.89 x	42.64 x	26.89	1290.60