Coveo

This dataset was released for the purpose of shopper intent prediction from clickstream data as part of the SIGIR Ecom 2021 Challenge. The dataset consists of user sessions using 30 minutes as the session gap threshold. The data was collected by Coveo from one of it’s partner’s e-commerce site. There are five types of events: add, click, detail, purchase and remove. Detail events correspond to the user visiting the detail page of an item (similarly to view events in e.g. Yoochoose). We use only these detail events to build the sessions.

Recall@20

BPR-Max

Cross-Entropy

MRR@20

BPR-Max

Cross-Entropy

GRU4REC-pytorch

Metric

BPR-Max

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.0501	0.0501	0.1464	0.0835	0.2172	0.0928	0.3123	0.0994
GRU4Rec Official	GRU4REC-pytorch params	0.0297	0.0297	0.0965	0.0525	0.1485	0.0593	0.2216	0.0643
GRU4REC-pytorch	OOB	0.0110	0.0110	0.0421	0.0212	0.0699	0.0248	0.1141	0.0279
GRU4REC-pytorch	OOB Correct Eval	0.0140	0.0140	0.0567	0.0281	0.0934	0.0329	0.1510	0.0368
GRU4REC-pytorch	Correct full	0.0165	0.0165	0.0626	0.0320	0.1014	0.0371	0.1582	0.0410

Cross-Entropy

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.0489	0.0489	0.1418	0.0814	0.2085	0.0901	0.2947	0.0960
GRU4Rec Official	GRU4REC-pytorch params	0.0275	0.0275	0.0871	0.0478	0.1362	0.0543	0.2061	0.0591
GRU4REC-pytorch	OOB	0.0312	0.0312	0.0468	0.0370	0.0537	0.0379	0.0628	0.0386
GRU4REC-pytorch	OOB Correct Eval	0.0315	0.0315	0.0487	0.0380	0.0566	0.0390	0.0663	0.0397
GRU4REC-pytorch	Correct full	0.0222	0.0222	0.0742	0.0397	0.1175	0.0454	0.1806	0.0497

Metric Diff

BPR-Max

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	GRU4REC-pytorch params	-40.78%	-40.78%	-34.09%	-37.15%	-31.64%	-36.10%	-29.03%	-35.25%
GRU4REC-pytorch	OOB	-78.15%	-78.15%	-71.20%	-74.56%	-67.80%	-73.24%	-63.46%	-71.96%
GRU4REC-pytorch	OOB Correct Eval	-72.16%	-72.16%	-61.23%	-66.35%	-57.01%	-64.60%	-51.65%	-63.00%
GRU4REC-pytorch	Correct full	-67.09%	-67.09%	-57.23%	-61.67%	-53.33%	-60.04%	-49.33%	-58.77%

Cross-Entropy

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	GRU4REC-pytorch params	-43.74%	-43.74%	-38.54%	-41.18%	-34.70%	-39.78%	-30.08%	-38.46%
GRU4REC-pytorch	OOB	-36.22%	-36.22%	-66.97%	-54.48%	-74.23%	-57.91%	-78.70%	-59.84%
GRU4REC-pytorch	OOB Correct Eval	-35.53%	-35.53%	-65.62%	-53.30%	-72.86%	-56.69%	-77.50%	-58.65%
GRU4REC-pytorch	Correct full	-54.53%	-54.53%	-47.65%	-51.22%	-43.66%	-49.61%	-38.73%	-48.21%

Hyperparameters

BPR-Max

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	GRU4REC-pytorch	GRU4REC-pytorch	GRU4REC-pytorch
Variant	Best params	GRU4REC-pytorch params	OOB	OOB Correct Eval	Correct full
loss	bpr-max	bpr-max	bpr-max	bpr-max	bpr-max
optim	adagrad	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False	False
embedding	0	512	512	512	512
final_act	elu-1	elu-1	elu-1	elu-1	elu-1
layers	512	512	512	512	512
batch_size	144	144	144	144	144
dropout_p_embed	0.35	0.35	N/A	N/A	0.35
dropout_p_hidden	0	0	N/A	N/A	0
learning_rate	0.05	0.05	0.05	0.05	0.05
momentum	0.4	0	N/A	N/A	N/A
n_sample	2048	0	N/A	N/A	N/A
sample_alpha	0.2	0	N/A	N/A	N/A
bpreg	1.85	0	N/A	N/A	N/A
logq	0	0	N/A	N/A	N/A

Cross-Entropy

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	GRU4REC-pytorch	GRU4REC-pytorch	GRU4REC-pytorch
Variant	Best params	GRU4REC-pytorch params	OOB	OOB Correct Eval	Correct full
loss	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy
optim	adagrad	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False	False
embedding	0	512	512	512	512
final_act	softmax	softmax	softmax	softmax	softmax
layers	512	512	512	512	512
batch_size	32	32	32	32	32
dropout_p_embed	0.4	0.4	N/A	N/A	0.4
dropout_p_hidden	0.15	0.15	N/A	N/A	0.15
learning_rate	0.03	0.03	0.03	0.03	0.03
momentum	0	0	N/A	N/A	N/A
n_sample	2048	0	N/A	N/A	N/A
sample_alpha	0	0	N/A	N/A	N/A
bpreg	0	0	N/A	N/A	N/A
logq	1	0	N/A	N/A	N/A

Runtimes

BPR-Max

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	12.38			704.14	100615.00
GRU4Rec Official	GRU4REC-pytorch params	9.25	0.75 x		941.96	134690.00
GRU4REC-pytorch	OOB	23.35	1.89 x	2.52 x	370.06	53288.62
GRU4REC-pytorch	OOB Correct Eval	23.19	1.87 x	2.51 x	372.64	53660.93
GRU4REC-pytorch	Correct full	31.91	2.58 x	3.45 x	271.16	39046.98

Cross-Entropy

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	37.17			1047.47	33505.00
GRU4Rec Official	GRU4REC-pytorch params	30.51	0.82 x		1276.12	40819.00
GRU4REC-pytorch	OOB	85.06	2.29 x	2.79 x	457.51	14640.34
GRU4REC-pytorch	OOB Correct Eval	86.98	2.34 x	2.85 x	447.82	14330.19
GRU4REC-pytorch	Correct full	89.90	2.42 x	2.95 x	432.89	13852.60

Torch-GRU4Rec

Metric

BPR-Max

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.0501	0.0501	0.1464	0.0835	0.2172	0.0928	0.3123	0.0994
GRU4Rec Official	Torch-GRU4Rec params	0.0484	0.0484	0.1430	0.0811	0.2107	0.0901	0.2995	0.0962
Torch-GRU4Rec	OOB	0.0479	0.0479	0.1400	0.0794	0.2067	0.0883	0.2960	0.0944

Cross-Entropy

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.0489	0.0489	0.1418	0.0814	0.2085	0.0901	0.2947	0.0960
GRU4Rec Official	Torch-GRU4Rec params	0.0473	0.0473	0.1379	0.0786	0.2015	0.0871	0.2848	0.0928
Torch-GRU4Rec	OOB	0.0445	0.0445	0.1343	0.0755	0.1949	0.0835	0.2835	0.0897

Metric Diff

BPR-Max

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	Torch-GRU4Rec params	-3.52%	-3.52%	-2.27%	-2.86%	-2.98%	-2.99%	-4.09%	-3.21%
Torch-GRU4Rec	OOB	-4.43%	-4.43%	-4.35%	-4.88%	-4.84%	-4.92%	-5.23%	-5.01%

Cross-Entropy

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	Torch-GRU4Rec params	-3.29%	-3.29%	-2.74%	-3.34%	-3.35%	-3.38%	-3.37%	-3.37%
Torch-GRU4Rec	OOB	-8.98%	-8.98%	-5.30%	-7.14%	-6.55%	-7.33%	-3.80%	-6.61%

Hyperparameters

BPR-Max

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	Torch-GRU4Rec
Variant	Best params	Torch-GRU4Rec params	OOB
loss	bpr-max	bpr-max	bpr-max
optim	adagrad	adagrad	adagrad
constrained_embedding	True	False	False
embedding	0	512	512
final_act	elu-1	elu-1	elu-1
layers	512	512	512
batch_size	144	144	144
dropout_p_embed	0.35	0.35	0.35
dropout_p_hidden	0	0	0
learning_rate	0.05	0.05	0.05
momentum	0.4	0	N/A
n_sample	2048	2048	2048
sample_alpha	0.2	0.2	0.2
bpreg	1.85	1.85	1.85
logq	0	0	N/A

Cross-Entropy

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	Torch-GRU4Rec
Variant	Best params	Torch-GRU4Rec params	OOB
loss	cross-entropy	cross-entropy	cross-entropy
optim	adagrad	adagrad	adagrad
constrained_embedding	True	False	False
embedding	0	512	512
final_act	softmax	softmax	softmax
layers	512	512	512
batch_size	32	32	32
dropout_p_embed	0.4	0.4	0.4
dropout_p_hidden	0.15	0.15	0.15
learning_rate	0.03	0.03	0.03
momentum	0	0	N/A
n_sample	2048	2048	2048
sample_alpha	0	0	0
bpreg	0	0	0
logq	1	0	N/A

Runtimes

BPR-Max

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	12.38			704.14	100615.00
GRU4Rec Official	Torch-GRU4Rec params	12.23	0.99 x		712.45	101803.00
Torch-GRU4Rec	OOB	32.18	2.60 x	2.63 x	270.97	38719.00

Cross-Entropy

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	37.17			1047.47	33505.00
GRU4Rec Official	Torch-GRU4Rec params	36.81	0.99 x		1057.66	33830.00
Torch-GRU4Rec	OOB	91.28	2.46 x	2.48 x	426.78	13650.91

Recpack

Metric

BPR-Max

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.0501	0.0501	0.1464	0.0835	0.2172	0.0928	0.3123	0.0994
GRU4Rec Official	Recpack params	0.0471	0.0471	0.1418	0.0800	0.2104	0.0890	0.3003	0.0951
Recpack	OOB	0.0287	0.0287	0.0905	0.0499	0.1405	0.0565	0.2091	0.0612
Recpack	Correct exp	0.0265	0.0265	0.0855	0.0469	0.1309	0.0528	0.1953	0.0573
Recpack	Correct full	0.0205	0.0205	0.0643	0.0351	0.1005	0.0399	0.1536	0.0435

Cross-Entropy

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.0489	0.0489	0.1418	0.0814	0.2085	0.0901	0.2947	0.0960
GRU4Rec Official	Recpack params	0.0458	0.0458	0.1359	0.0770	0.2009	0.0856	0.2829	0.0912
GRU4Rec Official	Recpack params	0.0483	0.0483	0.1368	0.0788	0.2012	0.0873	0.2851	0.0931
Recpack	OOB	0.0381	0.0381	0.1108	0.0635	0.1635	0.0704	0.2335	0.0752
Recpack	Correct exp	0.0375	0.0375	0.1094	0.0622	0.1605	0.0689	0.2323	0.0739
Recpack	Correct full	0.0365	0.0365	0.1061	0.0603	0.1596	0.0673	0.2309	0.0722

Metric Diff

BPR-Max

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	Recpack params	-6.02%	-6.02%	-3.08%	-4.21%	-3.16%	-4.15%	-3.84%	-4.25%
Recpack	OOB	-42.83%	-42.83%	-38.18%	-40.25%	-35.29%	-39.19%	-33.04%	-38.46%
Recpack	Correct exp	-47.19%	-47.19%	-41.56%	-43.86%	-39.75%	-43.08%	-37.45%	-42.37%
Recpack	Correct full	-59.09%	-59.09%	-56.05%	-57.94%	-53.73%	-57.07%	-50.80%	-56.25%

Cross-Entropy

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	Recpack params	-6.26%	-6.26%	-4.13%	-5.39%	-3.64%	-5.06%	-4.01%	-5.02%
GRU4Rec Official	Recpack params	-1.28%	-1.28%	-3.50%	-3.08%	-3.53%	-3.14%	-3.27%	-3.07%
Recpack	OOB	-22.07%	-22.07%	-21.87%	-22.00%	-21.61%	-21.88%	-20.77%	-21.64%
Recpack	Correct exp	-23.40%	-23.40%	-22.85%	-23.52%	-23.04%	-23.50%	-21.17%	-23.06%
Recpack	Correct full	-25.41%	-25.41%	-25.15%	-25.92%	-23.45%	-25.33%	-21.65%	-24.80%

Hyperparameters

BPR-Max

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	Recpack	Recpack	Recpack
Variant	Best params	Recpack params	OOB	Correct exp	Correct full
loss	bpr-max	bpr-max	bpr-max	bpr-max	bpr-max
optim	adagrad	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False	False
embedding	0	512	512	512	512
final_act	elu-1	linear	N/A	N/A	N/A
layers	512	512	512	512	512
batch_size	144	144	144	144	144
dropout_p_embed	0.35	0.35	0.35	0.35	0.35
dropout_p_hidden	0	0	0.35	0.35	0
learning_rate	0.05	0.05	0.05	0.05	0.05
momentum	0.4	0	N/A	N/A	N/A
n_sample	2048	2048	2048	2048	2048
sample_alpha	0.2	0	N/A	N/A	N/A
bpreg	1.85	1.85	1	1.85	1.85
logq	0	0	N/A	N/A	N/A

Cross-Entropy

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	GRU4Rec Official	Recpack	Recpack	Recpack
Variant	Best params	Recpack params	Recpack params	OOB	Correct exp	Correct full
loss	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy
optim	adagrad	adagrad	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False	False	False
embedding	0	512	512	512	512	512
final_act	softmax	softmax	softmax	softmax	softmax	softmax
layers	512	512	512	512	512	512
batch_size	32	32	32	32	32	32
dropout_p_embed	0.4	0.4	0.4	0.4	0.4	0.4
dropout_p_hidden	0.15	0.15	0.15	0.4	0.4	0.15
learning_rate	0.03	0.03	0.03	0.03	0.03	0.03
momentum	0	0	N/A	N/A	N/A	N/A
n_sample	2048	2048	ALL	N/A	N/A	N/A
sample_alpha	0	0	N/A	N/A	N/A	N/A
bpreg	0	0	0	0	0	0
logq	1	0	N/A	N/A	N/A	N/A

Runtimes

BPR-Max

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	12.38			704.14	100615.00
GRU4Rec Official	Recpack params	12.19	0.98 x		715.00	102167.00
Recpack	OOB	770.91	62.27 x	63.24 x	110.48	1615.97
Recpack	Correct exp	760.89	61.46 x	62.42 x	111.91	1636.88
Recpack	Correct full	758.63	61.28 x	62.23 x	112.25	1641.79

Cross-Entropy

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	37.17			1047.47	33505.00
GRU4Rec Official	Recpack params	36.65	0.99 x		1062.34	33980.00
GRU4Rec Official	Recpack params	74.28	2.00 x	2.03 x	524.23	16768.00
Recpack	OOB	662.69	17.83 x	18.08 x	346.18	1881.49
Recpack	Correct exp	656.08	17.65 x	17.90 x	349.57	1899.95
Recpack	Correct full	637.93	17.16 x	17.41 x	359.21	1952.35

GRU4Rec_Tensorflow

Metric

BPR-Max

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Cross-Entropy

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.0489	0.0489	0.1418	0.0814	0.2085	0.0901	0.2947	0.0960
GRU4Rec Official	GRU4Rec_Tensorflow params	0.0273	0.0273	0.0853	0.0468	0.1326	0.0530	0.2001	0.0576
GRU4Rec_Tensorflow	OOB	0.0025	0.0025	0.0103	0.0050	0.0204	0.0063	0.0375	0.0075
GRU4Rec_Tensorflow	Correct Exp	0.0252	0.0252	0.0804	0.0441	0.1276	0.0503	0.1951	0.0549

Metric Diff

BPR-Max

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Cross-Entropy

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	GRU4Rec_Tensorflow params	-44.10%	-44.10%	-39.85%	-42.52%	-36.42%	-41.24%	-32.12%	-40.06%
GRU4Rec_Tensorflow	OOB	-94.84%	-94.84%	-92.70%	-93.88%	-90.24%	-93.01%	-87.27%	-92.22%
GRU4Rec_Tensorflow	Correct Exp	-48.49%	-48.49%	-43.33%	-45.79%	-38.80%	-44.18%	-33.81%	-42.79%

Hyperparameters

BPR-Max

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Cross-Entropy

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	GRU4Rec_Tensorflow	GRU4Rec_Tensorflow
Variant	Best params	GRU4Rec_Tensorflow params	OOB	Correct Exp
loss	cross-entropy	cross-entropy	cross-entropy	cross-entropy
optim	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False
embedding	0	512	512	512
final_act	softmax	softmax	softmax	softmax
layers	512	512	512	512
batch_size	32	32	50	32
dropout_p_embed	0.4	0	N/A	N/A
dropout_p_hidden	0.15	0.15	0.15	0.15
learning_rate	0.03	0.03	0.03	0.03
momentum	0	0	N/A	N/A
n_sample	2048	0	N/A	N/A
sample_alpha	0	0	N/A	N/A
bpreg	0	0	N/A	N/A
logq	1	0	N/A	N/A

Runtimes

BPR-Max

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Cross-Entropy

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	37.17			1047.47	33505.00
GRU4Rec Official	GRU4Rec_Tensorflow params	29.40	0.79 x		1324.38	42363.00
GRU4Rec_Tensorflow	OOB	55.39	1.49 x	1.88 x	449.67	22483.60
GRU4Rec_Tensorflow	Correct Exp	81.90	2.20 x	2.79 x	475.29	15209.23

KerasGRU4Rec

Metric

BPR-Max

Note

BPR-Max is not supported by KerasGRU4Rec

Cross-Entropy

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.0489	0.0489	0.1418	0.0814	0.2085	0.0901	0.2947	0.0960
GRU4Rec Official	KerasGRU4Rec params	0.0466	0.0466	0.1360	0.0775	0.1993	0.0859	0.2852	0.0917
GRU4Rec Official	KerasGRU4Rec params	0.0480	0.0480	0.1383	0.0789	0.2029	0.0875	0.2880	0.0933
KerasGRU4Rec	OOB	0.0451	0.0451	0.1270	0.0732	0.1888	0.0813	0.2739	0.0871
KerasGRU4Rec	Correct exp	0.0477	0.0477	0.1370	0.0786	0.1997	0.0868	0.2822	0.0925
KerasGRU4Rec	Correct full	0.0469	0.0469	0.1361	0.0778	0.1993	0.0861	0.2826	0.0918

Metric Diff

BPR-Max

Note

BPR-Max is not supported by KerasGRU4Rec

Cross-Entropy

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	KerasGRU4Rec params	-4.62%	-4.62%	-4.11%	-4.76%	-4.40%	-4.71%	-3.22%	-4.46%
GRU4Rec Official	KerasGRU4Rec params	-1.87%	-1.87%	-2.47%	-2.96%	-2.68%	-2.95%	-2.27%	-2.82%
KerasGRU4Rec	OOB	-7.72%	-7.72%	-10.47%	-10.03%	-9.44%	-9.81%	-7.06%	-9.24%
KerasGRU4Rec	Correct exp	-2.40%	-2.40%	-3.42%	-3.42%	-4.22%	-3.65%	-4.25%	-3.65%
KerasGRU4Rec	Correct full	-4.05%	-4.05%	-3.99%	-4.41%	-4.42%	-4.47%	-4.10%	-4.36%

Hyperparameters

BPR-Max

Note

BPR-Max is not supported by KerasGRU4Rec

Cross-Entropy

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	GRU4Rec Official	KerasGRU4Rec	KerasGRU4Rec	KerasGRU4Rec
Variant	Best params	KerasGRU4Rec params	KerasGRU4Rec params	OOB	Correct exp	Correct full
loss	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy
optim	adagrad	adagrad	adagrad	adam	adagrad	adagrad
constrained_embedding	True	False	False	False	False	False
embedding	0	0	0	0	0	0
final_act	softmax	softmax	softmax	softmax	softmax	softmax
layers	512	512	512	100	512	512
batch_size	32	32	32	32	32	32
dropout_p_embed	0.4	0	0	N/A	N/A	N/A
dropout_p_hidden	0.15	0.15	0.15	0.15	0.15	0.15
learning_rate	0.03	0.03	0.03	0.001	0.03	0.03
momentum	0	0	0	N/A	N/A	N/A
n_sample	2048	2048	ALL	N/A	N/A	N/A
sample_alpha	0	0	N/A	N/A	N/A	N/A
bpreg	0	0	0	N/A	N/A	N/A
logq	1	0	N/A	N/A	N/A	N/A

Runtimes

BPR-Max

Note

BPR-Max is not supported by KerasGRU4Rec

Cross-Entropy

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	37.17			1047.47	33505.00
GRU4Rec Official	KerasGRU4Rec params	31.86	0.86 x		1222.20	39093.00
GRU4Rec Official	KerasGRU4Rec params	68.66	1.85 x	2.16 x	567.11	18139.00
KerasGRU4Rec	OOB	568.22	15.29 x	17.83 x	68.49	2191.59
KerasGRU4Rec	Correct exp	677.80	18.24 x	21.27 x	57.41	1837.26
KerasGRU4Rec	Correct full	644.59	17.34 x	20.23 x	60.38	1932.25