Retailrocket

The dataset consists of view, add to cart, and transaction events collected over 4.5 months. We only keep the view events. The dataset was published to motivate research in the field of recommender systems.

Recall@20

BPR-Max

Cross-Entropy

MRR@20

BPR-Max

Cross-Entropy

GRU4REC-pytorch

Metric

BPR-Max

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.1224	0.1224	0.3196	0.1928	0.4181	0.2060	0.5187	0.2131
GRU4Rec Official	GRU4REC-pytorch params	0.0560	0.0560	0.1595	0.0916	0.2294	0.1009	0.3142	0.1067
GRU4REC-pytorch	OOB	0.0096	0.0096	0.0355	0.0184	0.0544	0.0210	0.0794	0.0227
GRU4REC-pytorch	OOB Correct Eval	0.0406	0.0406	0.1341	0.0728	0.1933	0.0807	0.2528	0.0848
GRU4REC-pytorch	Correct full	0.0456	0.0456	0.1443	0.0804	0.1976	0.0875	0.2554	0.0916

Cross-Entropy

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.1150	0.1150	0.3026	0.1814	0.4019	0.1947	0.4953	0.2012
GRU4Rec Official	GRU4REC-pytorch params	0.0578	0.0578	0.1681	0.0955	0.2441	0.1056	0.3378	0.1121
GRU4REC-pytorch	OOB	0.0479	0.0479	0.0561	0.0511	0.0590	0.0515	0.0616	0.0517
GRU4REC-pytorch	OOB Correct Eval	0.0510	0.0510	0.0582	0.0539	0.0607	0.0543	0.0642	0.0545
GRU4REC-pytorch	Correct full	0.0492	0.0492	0.1544	0.0858	0.2163	0.0941	0.2812	0.0986

Metric Diff

BPR-Max

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	GRU4REC-pytorch params	-54.23%	-54.23%	-50.10%	-52.48%	-45.13%	-51.04%	-39.41%	-49.92%
GRU4REC-pytorch	OOB	-92.12%	-92.12%	-88.90%	-90.45%	-86.99%	-89.83%	-84.69%	-89.36%
GRU4REC-pytorch	OOB Correct Eval	-66.88%	-66.88%	-58.06%	-62.23%	-53.78%	-60.84%	-51.27%	-60.18%
GRU4REC-pytorch	Correct full	-62.76%	-62.76%	-54.84%	-58.30%	-52.74%	-57.52%	-50.76%	-57.01%

Cross-Entropy

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	GRU4REC-pytorch params	-49.71%	-49.71%	-44.43%	-47.36%	-39.25%	-45.78%	-31.79%	-44.32%
GRU4REC-pytorch	OOB	-58.36%	-58.36%	-81.47%	-71.82%	-85.31%	-73.53%	-87.55%	-74.30%
GRU4REC-pytorch	OOB Correct Eval	-55.66%	-55.66%	-80.76%	-70.27%	-84.90%	-72.13%	-87.05%	-72.91%
GRU4REC-pytorch	Correct full	-57.19%	-57.19%	-48.99%	-52.73%	-46.19%	-51.69%	-43.23%	-51.02%

Hyperparameters

BPR-Max

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	GRU4REC-pytorch	GRU4REC-pytorch	GRU4REC-pytorch
Variant	Best params	GRU4REC-pytorch params	OOB	OOB Correct Eval	Correct full
loss	bpr-max	bpr-max	bpr-max	bpr-max	bpr-max
optim	adagrad	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False	False
embedding	0	224	224	224	224
final_act	elu-0.5	elu-0.5	elu-0.5	elu-0.5	elu-0.5
layers	224	224	224	224	224
batch_size	80	80	80	80	80
dropout_p_embed	0.5	0.5	N/A	N/A	0.5
dropout_p_hidden	0.05	0.05	N/A	N/A	0.05
learning_rate	0.05	0.05	0.05	0.05	0.05
momentum	0.4	0	N/A	N/A	N/A
n_sample	2048	0	N/A	N/A	N/A
sample_alpha	0.4	0	N/A	N/A	N/A
bpreg	1.95	0	N/A	N/A	N/A
logq	0	0	N/A	N/A	N/A

Cross-Entropy

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	GRU4REC-pytorch	GRU4REC-pytorch	GRU4REC-pytorch
Variant	Best params	GRU4REC-pytorch params	OOB	OOB Correct Eval	Correct full
loss	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy
optim	adagrad	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False	False
embedding	0	192	192	192	192
final_act	softmax	softmax	softmax	softmax	softmax
layers	192	192	192	192	192
batch_size	240	240	240	240	240
dropout_p_embed	0.5	0.5	N/A	N/A	0.5
dropout_p_hidden	0.05	0.05	N/A	N/A	0.05
learning_rate	0.085	0.085	0.085	0.085	0.085
momentum	0.3	0	N/A	N/A	N/A
n_sample	2048	0	N/A	N/A	N/A
sample_alpha	0.3	0	N/A	N/A	N/A
bpreg	0	0	N/A	N/A	N/A
logq	1	0	N/A	N/A	N/A

Runtimes

BPR-Max

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	6.86			1019.34	80807.00
GRU4Rec Official	GRU4REC-pytorch params	5.22	0.76 x		1338.98	106326.00
GRU4REC-pytorch	OOB	20.53	2.99 x	3.93 x	337.68	27014.01
GRU4REC-pytorch	OOB Correct Eval	20.44	2.98 x	3.92 x	339.01	27120.93
GRU4REC-pytorch	Correct full	24.42	3.56 x	4.68 x	283.82	22705.56

Cross-Entropy

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	2.77			880.71	199935.00
GRU4Rec Official	GRU4REC-pytorch params	1.89	0.68 x		1279.60	293600.00
GRU4REC-pytorch	OOB	10.10	3.65 x	5.34 x	228.56	54855.21
GRU4REC-pytorch	OOB Correct Eval	10.03	3.62 x	5.31 x	230.11	55225.62
GRU4REC-pytorch	Correct full	9.14	3.30 x	4.84 x	252.47	60592.49

Torch-GRU4Rec

Metric

BPR-Max

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.1224	0.1224	0.3196	0.1928	0.4181	0.2060	0.5187	0.2131
GRU4Rec Official	Torch-GRU4Rec params	0.0979	0.0979	0.2672	0.1576	0.3631	0.1703	0.4639	0.1773
Torch-GRU4Rec	OOB	0.0926	0.0926	0.2413	0.1451	0.3336	0.1574	0.4201	0.1633

Cross-Entropy

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.1150	0.1150	0.3026	0.1814	0.4019	0.1947	0.4953	0.2012
GRU4Rec Official	Torch-GRU4Rec params	0.0907	0.0907	0.2462	0.1456	0.3331	0.1571	0.4249	0.1635
Torch-GRU4Rec	OOB	0.0806	0.0806	0.2206	0.1300	0.2994	0.1405	0.3834	0.1464

Metric Diff

BPR-Max

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	Torch-GRU4Rec params	-20.04%	-20.04%	-16.41%	-18.23%	-13.15%	-17.33%	-10.56%	-16.77%
Torch-GRU4Rec	OOB	-24.35%	-24.35%	-24.51%	-24.70%	-20.22%	-23.61%	-19.00%	-23.35%

Cross-Entropy

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	Torch-GRU4Rec params	-21.12%	-21.12%	-18.63%	-19.73%	-17.11%	-19.29%	-14.20%	-18.73%
Torch-GRU4Rec	OOB	-29.89%	-29.89%	-27.10%	-28.36%	-25.50%	-27.83%	-22.60%	-27.27%

Hyperparameters

BPR-Max

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	Torch-GRU4Rec
Variant	Best params	Torch-GRU4Rec params	OOB
loss	bpr-max	bpr-max	bpr-max
optim	adagrad	adagrad	adagrad
constrained_embedding	True	False	False
embedding	0	224	224
final_act	elu-0.5	elu-0.5	elu-0.5
layers	224	224	224
batch_size	80	80	80
dropout_p_embed	0.5	0.5	0.5
dropout_p_hidden	0.05	0.05	0.05
learning_rate	0.05	0.05	0.05
momentum	0.4	0	N/A
n_sample	2048	2048	2048
sample_alpha	0.4	0.4	0.4
bpreg	1.95	1.95	1.95
logq	0	0	N/A

Cross-Entropy

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	Torch-GRU4Rec
Variant	Best params	Torch-GRU4Rec params	OOB
loss	cross-entropy	cross-entropy	cross-entropy
optim	adagrad	adagrad	adagrad
constrained_embedding	True	False	False
embedding	0	192	192
final_act	softmax	softmax	softmax
layers	192	192	192
batch_size	240	240	240
dropout_p_embed	0.5	0.5	0.5
dropout_p_hidden	0.05	0.05	0.05
learning_rate	0.085	0.085	0.085
momentum	0.3	0	N/A
n_sample	2048	2048	2048
sample_alpha	0.3	0.3	0.3
bpreg	0	0	0
logq	1	0	N/A

Runtimes

BPR-Max

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	6.86			1019.34	80807.00
GRU4Rec Official	Torch-GRU4Rec params	6.53	0.95 x		1071.09	84909.00
Torch-GRU4Rec	OOB	26.39	3.85 x	4.04 x	265.19	21022.30

Cross-Entropy

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	2.77			880.71	199935.00
GRU4Rec Official	Torch-GRU4Rec params	2.45	0.88 x		996.78	226283.00
Torch-GRU4Rec	OOB	10.67	3.85 x	4.36 x	229.70	52144.58

Recpack

Metric

BPR-Max

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.1224	0.1224	0.3196	0.1928	0.4181	0.2060	0.5187	0.2131
GRU4Rec Official	Recpack params	0.0883	0.0883	0.2491	0.1449	0.3449	0.1577	0.4455	0.1647
Recpack	OOB	0.0618	0.0618	0.1757	0.1010	0.2461	0.1104	0.3226	0.1157
Recpack	Correct exp	0.0567	0.0567	0.1667	0.0954	0.2320	0.1041	0.3054	0.1092
Recpack	Correct full	0.0641	0.0641	0.1902	0.1082	0.2679	0.1185	0.3550	0.1245

Cross-Entropy

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.1150	0.1150	0.3026	0.1814	0.4019	0.1947	0.4953	0.2012
GRU4Rec Official	Recpack params	0.0914	0.0914	0.2458	0.1457	0.3356	0.1577	0.4235	0.1637
GRU4Rec Official	Recpack params	0.1040	0.1040	0.2670	0.1620	0.3508	0.1732	0.4322	0.1788
Recpack	OOB	0.0609	0.0609	0.1514	0.0935	0.1960	0.0995	0.2442	0.1029
Recpack	Correct exp	0.0545	0.0545	0.1390	0.0845	0.1780	0.0897	0.2210	0.0927
Recpack	Correct full	0.0650	0.0650	0.1755	0.1044	0.2323	0.1119	0.2866	0.1157

Metric Diff

BPR-Max

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	Recpack params	-27.88%	-27.88%	-22.07%	-24.82%	-17.51%	-23.47%	-14.11%	-22.71%
Recpack	OOB	-49.51%	-49.51%	-45.04%	-47.59%	-41.14%	-46.42%	-37.80%	-45.71%
Recpack	Correct exp	-53.65%	-53.65%	-47.84%	-50.53%	-44.51%	-49.50%	-41.12%	-48.76%
Recpack	Correct full	-47.60%	-47.60%	-40.50%	-43.85%	-35.92%	-42.49%	-31.56%	-41.58%

Cross-Entropy

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	Recpack params	-20.58%	-20.58%	-18.79%	-19.70%	-16.48%	-19.03%	-14.48%	-18.64%
GRU4Rec Official	Recpack params	-9.63%	-9.63%	-11.77%	-10.72%	-12.71%	-11.05%	-12.74%	-11.14%
Recpack	OOB	-47.01%	-47.01%	-49.97%	-48.46%	-51.22%	-48.90%	-50.69%	-48.89%
Recpack	Correct exp	-52.61%	-52.61%	-54.07%	-53.44%	-55.70%	-53.94%	-55.38%	-53.95%
Recpack	Correct full	-43.53%	-43.53%	-41.99%	-42.48%	-42.20%	-42.51%	-42.14%	-42.51%

Hyperparameters

BPR-Max

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	Recpack	Recpack	Recpack
Variant	Best params	Recpack params	OOB	Correct exp	Correct full
loss	bpr-max	bpr-max	bpr-max	bpr-max	bpr-max
optim	adagrad	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False	False
embedding	0	224	224	224	224
final_act	elu-0.5	linear	N/A	N/A	N/A
layers	224	224	224	224	224
batch_size	80	80	80	80	80
dropout_p_embed	0.5	0.5	0.5	0.5	0.5
dropout_p_hidden	0.05	0.05	0.5	0.5	0.05
learning_rate	0.05	0.05	0.05	0.05	0.05
momentum	0.4
n_sample	2048	2048	2048	2048	2048
sample_alpha	0.4	0	N/A	N/A	N/A
bpreg	1.95	1.95	1	1.95	1.95
logq	0	0	N/A	N/A	N/A

Cross-Entropy

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	GRU4Rec Official	Recpack	Recpack	Recpack
Variant	Best params	Recpack params	Recpack params	OOB	Correct exp	Correct full
loss	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy
optim	adagrad	adagrad	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False	False	False
embedding	0	192	192	192	192	192
final_act	softmax	softmax	softmax	softmax	softmax	softmax
layers	192	192	192	192	192	192
batch_size	240	240	240	240	240	240
dropout_p_embed	0.5	0.5	0.5	0.5	0.5	0.5
dropout_p_hidden	0.05	0.05	0.05	0.5	0.5	0.05
learning_rate	0.085	0.085	0.085	0.085	0.085	0.085
momentum	0.3	0	0	N/A	N/A	N/A
n_sample	2048	2048	ALL	N/A	N/A	N/A
sample_alpha	0.3	0	N/A	N/A	N/A	N/A
bpreg	0	0	0	0	0	0
logq	1	0	N/A	N/A	N/A	N/A

Runtimes

BPR-Max

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	6.86			1019.34	80807.00
GRU4Rec Official	Recpack params	6.43	0.94 x		1087.28	86193.00
Recpack	OOB	450.18	65.62 x	70.01 x	125.44	1232.31
Recpack	Correct exp	451.63	65.84 x	70.24 x	125.05	1228.42
Recpack	Correct full	461.24	67.24 x	71.73 x	122.40	1202.41

Cross-Entropy

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	2.77			880.71	199935.00
GRU4Rec Official	Recpack params	2.45	0.88 x		997.22	226383.00
GRU4Rec Official	Recpack params	72.09	26.03 x	29.42 x	33.89	7693.00
Recpack	OOB	122.27	44.14 x	49.91 x	253.53	4535.76
Recpack	Correct exp	126.13	45.53 x	51.48 x	245.85	4398.31
Recpack	Correct full	121.39	43.82 x	49.55 x	255.38	4568.79

GRU4Rec_Tensorflow

Metric

BPR-Max

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Cross-Entropy

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.1150	0.1150	0.3026	0.1814	0.4019	0.1947	0.4953	0.2012
GRU4Rec Official	GRU4Rec_Tensorflow params	0.0541	0.0541	0.1662	0.0931	0.2425	0.1031	0.3309	0.1092
GRU4Rec_Tensorflow	OOB	0.0130	0.0130	0.0328	0.0201	0.0381	0.0208	0.0439	0.0212
GRU4Rec_Tensorflow	Correct Exp	0.0612	0.0612	0.1776	0.1019	0.2541	0.1120	0.3388	0.1179

Metric Diff

BPR-Max

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Cross-Entropy

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	GRU4Rec_Tensorflow params	-52.99%	-52.99%	-45.08%	-48.71%	-39.67%	-47.05%	-33.19%	-45.73%
GRU4Rec_Tensorflow	OOB	-88.71%	-88.71%	-89.16%	-88.92%	-90.51%	-89.32%	-91.14%	-89.46%
GRU4Rec_Tensorflow	Correct Exp	-46.76%	-46.76%	-41.29%	-43.85%	-36.77%	-42.45%	-31.59%	-41.40%

Hyperparameters

BPR-Max

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Cross-Entropy

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	GRU4Rec_Tensorflow	GRU4Rec_Tensorflow
Variant	Best params	GRU4Rec_Tensorflow params	OOB	Correct Exp
loss	cross-entropy	cross-entropy	cross-entropy	cross-entropy
optim	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False
embedding	0	192	192	192
final_act	softmax	softmax	softmax	softmax
layers	192	192	192	192
batch_size	240	240	50	240
dropout_p_embed	0.5	0	N/A	N/A
dropout_p_hidden	0.05	0.05	0.05	0.05
learning_rate	0.085	0.085	0.085	0.085
momentum	0.3	0	N/A	N/A
n_sample	2048	0	N/A	N/A
sample_alpha	0.3	0	N/A	N/A
bpreg	0	0	N/A	N/A
logq	1	0	N/A	N/A

Runtimes

BPR-Max

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Cross-Entropy

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	2.77			880.71	199935.00
GRU4Rec Official	GRU4Rec_Tensorflow params	1.86	0.67 x		1298.62	297964.00
GRU4Rec_Tensorflow	OOB	22.97	8.29 x	12.35 x	482.83	24141.37
GRU4Rec_Tensorflow	Correct Exp	5.09	1.84 x	2.74 x	454.82	109157.42

KerasGRU4Rec

Metric

BPR-Max

Note

BPR-Max is not supported by KerasGRU4Rec

Cross-Entropy

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.1150	0.1150	0.3026	0.1814	0.4019	0.1947	0.4953	0.2012
GRU4Rec Official	KerasGRU4Rec params	0.0981	0.0981	0.2531	0.1535	0.3338	0.1643	0.4239	0.1705
GRU4Rec Official	KerasGRU4Rec params	0.1057	0.1057	0.2601	0.1608	0.3405	0.1715	0.4200	0.1771
KerasGRU4Rec	OOB	0.0713	0.0713	0.1899	0.1124	0.2520	0.1207	0.3151	0.1250
KerasGRU4Rec	Correct exp	0.0962	0.0962	0.2440	0.1488	0.3209	0.1591	0.3938	0.1642
KerasGRU4Rec	Correct full	0.0962	0.0962	0.2463	0.1496	0.3206	0.1595	0.3957	0.1647

Metric Diff

BPR-Max

Note

BPR-Max is not supported by KerasGRU4Rec

Cross-Entropy

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	KerasGRU4Rec params	-14.73%	-14.73%	-16.36%	-15.38%	-16.94%	-15.59%	-14.42%	-15.25%
GRU4Rec Official	KerasGRU4Rec params	-8.13%	-8.13%	-14.05%	-11.37%	-15.27%	-11.90%	-15.20%	-12.01%
KerasGRU4Rec	OOB	-37.98%	-37.98%	-37.26%	-38.05%	-37.30%	-38.03%	-36.38%	-37.88%
KerasGRU4Rec	Correct exp	-16.38%	-16.38%	-19.37%	-17.99%	-20.15%	-18.27%	-20.48%	-18.42%
KerasGRU4Rec	Correct full	-16.34%	-16.34%	-18.61%	-17.54%	-20.22%	-18.06%	-20.11%	-18.15%

Hyperparameters

BPR-Max

Note

BPR-Max is not supported by KerasGRU4Rec

Cross-Entropy

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	GRU4Rec Official	KerasGRU4Rec	KerasGRU4Rec	KerasGRU4Rec
Variant	Best params	KerasGRU4Rec params	KerasGRU4Rec params	OOB	Correct exp	Correct full
loss	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy
optim	adagrad	adagrad	adagrad	adam	adagrad	adagrad
constrained_embedding	True	False	False	False	False	False
embedding	0	0	0	0	0	0
final_act	softmax	softmax	softmax	softmax	softmax	softmax
layers	192	192	192	100	192	192
batch_size	240	240	240	240	240	240
dropout_p_embed	0.5	0	0	N/A	N/A	N/A
dropout_p_hidden	0.05	0.05	0.05	0.25	0.05	0.05
learning_rate	0.085	0.085	0.085	0.001	0.085	0.085
momentum	0.3	0	0	N/A	N/A	N/A
n_sample	2048	2048	ALL	N/A	N/A	N/A
sample_alpha	0.3	0	N/A	N/A	N/A	N/A
bpreg	0	0	0	N/A	N/A	N/A
logq	1	0	N/A	N/A	N/A	N/A

Runtimes

BPR-Max

Note

BPR-Max is not supported by KerasGRU4Rec

Cross-Entropy

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	2.77			880.71	199935.00
GRU4Rec Official	KerasGRU4Rec params	2.20	0.79 x		1111.09	252235.00
GRU4Rec Official	KerasGRU4Rec params	71.93	25.97 x	32.70 x	33.96	7710.00
KerasGRU4Rec	OOB	276.29	99.74 x	125.59 x	8.35	2004.12
KerasGRU4Rec	Correct exp	282.69	102.05 x	128.50 x	8.16	1958.63
KerasGRU4Rec	Correct full	281.92	101.78 x	128.15 x	8.18	1964.05