Diginetica

The dataset consists of user sessions collected over six months. The sessions were extracted from the search engine logs of an e-commerce site, and contain item page views that were preceded by search queries. The dataset doesn’t have exact timestamps, but each event has a property denoting the elapsed time since its session’s first query. The day of the first queries is also known. The dataset was released for the CIKM Cup 2016 Track 2: Personalized E-Commerce Search Challenge.

Recall@20

BPR-Max

Cross-Entropy

MRR@20

BPR-Max

Cross-Entropy

GRU4REC-pytorch

Metric

BPR-Max

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.0688	0.0688	0.2304	0.1237	0.3533	0.1399	0.4995	0.1500
GRU4Rec Official	GRU4REC-pytorch params	0.0365	0.0365	0.1283	0.0675	0.2066	0.0778	0.3141	0.0851
GRU4REC-pytorch	OOB	0.0006	0.0006	0.0020	0.0011	0.0039	0.0013	0.0070	0.0015
GRU4REC-pytorch	OOB Correct Eval	0.0239	0.0239	0.0937	0.0472	0.1537	0.0551	0.2367	0.0608
GRU4REC-pytorch	Correct full	0.0277	0.0277	0.1070	0.0543	0.1747	0.0632	0.2616	0.0692

Cross-Entropy

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.0647	0.0647	0.2220	0.1181	0.3414	0.1339	0.4874	0.1440
GRU4Rec Official	GRU4REC-pytorch params	0.0296	0.0296	0.1133	0.0576	0.1888	0.0675	0.2973	0.0749
GRU4REC-pytorch	OOB	0.0287	0.0287	0.0376	0.0321	0.0415	0.0327	0.0457	0.0329
GRU4REC-pytorch	OOB Correct Eval	0.0321	0.0321	0.0415	0.0357	0.0457	0.0363	0.0503	0.0366
GRU4REC-pytorch	Correct full	0.0288	0.0288	0.1135	0.0572	0.1860	0.0667	0.2862	0.0736

Metric Diff

BPR-Max

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	GRU4REC-pytorch params	-47.02%	-47.02%	-44.31%	-45.41%	-41.53%	-44.39%	-37.13%	-43.22%
GRU4REC-pytorch	OOB	-99.15%	-99.15%	-99.11%	-99.14%	-98.90%	-99.07%	-98.60%	-98.99%
GRU4REC-pytorch	OOB Correct Eval	-65.34%	-65.34%	-59.33%	-61.86%	-56.49%	-60.61%	-52.62%	-59.47%
GRU4REC-pytorch	Correct full	-59.74%	-59.74%	-53.55%	-56.08%	-50.55%	-54.81%	-47.62%	-53.87%

Cross-Entropy

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	GRU4REC-pytorch params	-54.28%	-54.28%	-48.96%	-51.20%	-44.68%	-49.54%	-39.00%	-47.98%
GRU4REC-pytorch	OOB	-55.59%	-55.59%	-83.08%	-72.79%	-87.86%	-75.61%	-90.62%	-77.13%
GRU4REC-pytorch	OOB Correct Eval	-50.44%	-50.44%	-81.29%	-69.74%	-86.62%	-72.89%	-89.68%	-74.58%
GRU4REC-pytorch	Correct full	-55.50%	-55.50%	-48.88%	-51.59%	-45.51%	-50.15%	-41.28%	-48.90%

Hyperparameters

BPR-Max

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	GRU4REC-pytorch	GRU4REC-pytorch	GRU4REC-pytorch
Variant	Best params	GRU4REC-pytorch params	OOB	OOB Correct Eval	Correct full
loss	bpr-max	bpr-max	bpr-max	bpr-max	bpr-max
optim	adagrad	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False	False
embedding	0	512	512	512	512
final_act	elu-1	elu-1	elu-1	elu-1	elu-1
layers	512	512	512	512	512
batch_size	128	128	128	128	128
dropout_p_embed	0.5	0.5	N/A	N/A	0.5
dropout_p_hidden	0.3	0.3	N/A	N/A	0.3
learning_rate	0.05	0.05	0.05	0.05	0.05
momentum	0.15	0	N/A	N/A	N/A
n_sample	2048	0	N/A	N/A	N/A
sample_alpha	0.3	0	N/A	N/A	N/A
bpreg	0.9	0	N/A	N/A	N/A
logq	0	0	N/A	N/A	N/A

Cross-Entropy

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	GRU4REC-pytorch	GRU4REC-pytorch	GRU4REC-pytorch
Variant	Best params	GRU4REC-pytorch params	OOB	OOB Correct Eval	Correct full
loss	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy
optim	adagrad	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False	False
embedding	0	192	192	192	192
final_act	softmax	softmax	softmax	softmax	softmax
layers	192	192	192	192	192
batch_size	128	128	128	128	128
dropout_p_embed	0.45	0.45	N/A	N/A	0.45
dropout_p_hidden	0.15	0.15	N/A	N/A	0.15
learning_rate	0.1	0.1	0.1	0.1	0.1
momentum	0	0	N/A	N/A	N/A
n_sample	2048	0	N/A	N/A	N/A
sample_alpha	0	0	N/A	N/A	N/A
bpreg	0	0	N/A	N/A	N/A
logq	1	0	N/A	N/A	N/A

Runtimes

BPR-Max

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	8.02			639.87	81757.00
GRU4Rec Official	GRU4REC-pytorch params	5.29	0.66 x		969.27	123869.00
GRU4REC-pytorch	OOB	32.24	4.02 x	6.09 x	158.86	20333.91
GRU4REC-pytorch	OOB Correct Eval	32.18	4.01 x	6.08 x	159.20	20378.12
GRU4REC-pytorch	Correct full	36.76	4.58 x	6.95 x	139.36	17838.75

Cross-Entropy

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	4.52			1134.52	144959.00
GRU4Rec Official	GRU4REC-pytorch params	3.67	0.81 x		1398.76	178755.00
GRU4REC-pytorch	OOB	17.65	3.90 x	4.81 x	290.27	37154.98
GRU4REC-pytorch	OOB Correct Eval	17.67	3.91 x	4.81 x	289.91	37108.16
GRU4REC-pytorch	Correct full	16.97	3.75 x	4.62 x	301.78	38627.84

Torch-GRU4Rec

Metric

BPR-Max

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.0688	0.0688	0.2304	0.1237	0.3533	0.1399	0.4995	0.1500
GRU4Rec Official	Torch-GRU4Rec params	0.0643	0.0643	0.2113	0.1143	0.3204	0.1287	0.4597	0.1383
Torch-GRU4Rec	OOB	0.0636	0.0636	0.2110	0.1137	0.3185	0.1278	0.4550	0.1372

Cross-Entropy

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.0647	0.0647	0.2220	0.1181	0.3414	0.1339	0.4874	0.1440
GRU4Rec Official	Torch-GRU4Rec params	0.0552	0.0552	0.1927	0.1020	0.2967	0.1157	0.4255	0.1246
Torch-GRU4Rec	OOB	0.0541	0.0541	0.1894	0.1002	0.2921	0.1137	0.4245	0.1229

Metric Diff

BPR-Max

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	Torch-GRU4Rec params	-6.58%	-6.58%	-8.30%	-7.53%	-9.32%	-7.97%	-7.97%	-7.76%
Torch-GRU4Rec	OOB	-7.64%	-7.64%	-8.42%	-8.08%	-9.85%	-8.63%	-8.92%	-8.54%

Cross-Entropy

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	Torch-GRU4Rec params	-14.78%	-14.78%	-13.17%	-13.69%	-13.08%	-13.55%	-12.70%	-13.47%
Torch-GRU4Rec	OOB	-16.45%	-16.45%	-14.67%	-15.19%	-14.44%	-15.04%	-12.90%	-14.71%

Hyperparameters

BPR-Max

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	Torch-GRU4Rec
Variant	Best params	Torch-GRU4Rec params	OOB
loss	bpr-max	bpr-max	bpr-max
optim	adagrad	adagrad	adagrad
constrained_embedding	True	False	False
embedding	0	512	512
final_act	elu-1	elu-1	elu-1
layers	512	512	512
batch_size	128	128	128
dropout_p_embed	0.5	0.5	0.5
dropout_p_hidden	0.3	0.3	0.3
learning_rate	0.05	0.05	0.05
momentum	0.15	0	N/A
n_sample	2048	2048	2048
sample_alpha	0.3	0.3	0.3
bpreg	0.9	0.9	0.9
logq	0	0	N/A

Cross-Entropy

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	Torch-GRU4Rec
Variant	Best params	Torch-GRU4Rec params	OOB
loss	cross-entropy	cross-entropy	cross-entropy
optim	adagrad	adagrad	adagrad
constrained_embedding	True	False	False
embedding	0	192	192
final_act	softmax	softmax	softmax
layers	192	192	192
batch_size	128	128	128
dropout_p_embed	0.45	0.45	0.45
dropout_p_hidden	0.15	0.15	0.15
learning_rate	0.1	0.1	0.1
momentum	0	0	N/A
n_sample	2048	2048	2048
sample_alpha	0	0	0
bpreg	0	0	0
logq	1	0	N/A

Runtimes

BPR-Max

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	8.02			639.87	81757.00
GRU4Rec Official	Torch-GRU4Rec params	7.71	0.96 x		665.42	85021.00
Torch-GRU4Rec	OOB	36.89	4.60 x	4.78 x	139.19	17783.97

Cross-Entropy

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	4.52			1134.52	144959.00
GRU4Rec Official	Torch-GRU4Rec params	4.48	0.99 x		1146.91	146542.00
Torch-GRU4Rec	OOB	17.74	3.92 x	3.96 x	289.38	36974.80

Recpack

Metric

BPR-Max

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.0688	0.0688	0.2304	0.1237	0.3533	0.1399	0.4995	0.1500
GRU4Rec Official	Recpack params	0.0635	0.0635	0.2101	0.1138	0.3203	0.1283	0.4586	0.1379
Recpack	OOB	0.0409	0.0409	0.1412	0.0752	0.2162	0.0851	0.3066	0.0913
Recpack	Correct exp	0.0449	0.0449	0.1609	0.0839	0.2558	0.0964	0.3744	0.1046
Recpack	Correct full	0.0430	0.0430	0.1643	0.0836	0.2580	0.0959	0.3756	0.1040

Cross-Entropy

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.0647	0.0647	0.2220	0.1181	0.3414	0.1339	0.4874	0.1440
GRU4Rec Official	Recpack params	0.0562	0.0562	0.1902	0.1019	0.2939	0.1157	0.4263	0.1248
GRU4Rec Official	Recpack params	0.0603	0.0603	0.2001	0.1083	0.3058	0.1223	0.4341	0.1312
Recpack	OOB	0.0411	0.0411	0.1442	0.0760	0.2238	0.0865	0.3279	0.0937
Recpack	Correct exp	0.0411	0.0411	0.1433	0.0757	0.2193	0.0857	0.3219	0.0927
Recpack	Correct full	0.0408	0.0408	0.1463	0.0766	0.2274	0.0873	0.3296	0.0944

Metric Diff

BPR-Max

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	Recpack params	-7.71%	-7.71%	-8.82%	-7.99%	-9.35%	-8.24%	-8.19%	-8.05%
Recpack	OOB	-40.57%	-40.57%	-38.70%	-39.21%	-38.82%	-39.16%	-38.63%	-39.10%
Recpack	Correct exp	-34.84%	-34.84%	-30.16%	-32.13%	-27.59%	-31.06%	-25.05%	-30.27%
Recpack	Correct full	-37.51%	-37.51%	-28.70%	-32.38%	-26.96%	-31.41%	-24.81%	-30.62%

Cross-Entropy

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	Recpack params	-13.23%	-13.23%	-14.32%	-13.69%	-13.90%	-13.59%	-12.52%	-13.34%
GRU4Rec Official	Recpack params	-6.88%	-6.88%	-9.87%	-8.32%	-10.41%	-8.62%	-10.92%	-8.93%
Recpack	OOB	-36.53%	-36.53%	-35.05%	-35.64%	-34.45%	-35.37%	-32.72%	-34.94%
Recpack	Correct exp	-36.50%	-36.50%	-35.44%	-35.94%	-35.75%	-36.00%	-33.96%	-35.62%
Recpack	Correct full	-37.00%	-37.00%	-34.07%	-35.11%	-33.38%	-34.75%	-32.37%	-34.46%

Hyperparameters

BPR-Max

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	Recpack	Recpack	Recpack
Variant	Best params	Recpack params	OOB	Correct exp	Correct full
loss	bpr-max	bpr-max	bpr-max	bpr-max	bpr-max
optim	adagrad	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False	False
embedding	0	512	512	512	512
final_act	elu-1	linear	N/A	N/A	N/A
layers	512	512	512	512	512
batch_size	128	128	128	128	128
dropout_p_embed	0.5	0.5	0.5	0.5	0.5
dropout_p_hidden	0.3	0.3	0.5	0.5	0.3
learning_rate	0.05	0.05	0.05	0.05	0.05
momentum	0.15	0	N/A	N/A	N/A
n_sample	2048	2048	2048	2048	2048
sample_alpha	0.3	0	N/A	N/A	N/A
bpreg	0.9	0.9	1	0.9	0.9
logq	0	0	N/A	N/A	N/A

Cross-Entropy

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	GRU4Rec Official	Recpack	Recpack	Recpack
Variant	Best params	Recpack params	Recpack params	OOB	Correct exp	Correct full
loss	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy
optim	adagrad	adagrad	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False	False	False
embedding	0	192	192	192	192	192
final_act	softmax	softmax	softmax	softmax	softmax	softmax
layers	192	192	192	192	192	192
batch_size	128	128	128	128	128	128
dropout_p_embed	0.45	0.45	0.45	0.45	0.45	0.45
dropout_p_hidden	0.15	0.15	0.15	0.45	0.45	0.15
learning_rate	0.1	0.1	0.1	0.1	0.1	0.1
momentum	0	0	0	N/A	N/A	N/A
n_sample	2048	2048	ALL	N/A	N/A	N/A
sample_alpha	0	0	N/A	N/A	N/A	N/A
bpreg	0	0	0	0	0	0
logq	1	0	N/A	N/A	N/A	N/A

Runtimes

BPR-Max

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	8.02			639.87	81757.00
GRU4Rec Official	Recpack params	7.73	0.96 x		663.79	84812.00
Recpack	OOB	352.17	43.91 x	45.56 x	79.68	1862.33
Recpack	Correct exp	346.70	43.23 x	44.85 x	80.94	1891.72
Recpack	Correct full	342.34	42.69 x	44.29 x	81.97	1915.79

Cross-Entropy

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	4.52			1134.52	144959.00
GRU4Rec Official	Recpack params	4.45	0.98 x		1154.05	147454.00
GRU4Rec Official	Recpack params	76.19	16.86 x	17.12 x	67.37	8608.00
Recpack	OOB	115.21	25.49 x	25.89 x	243.62	5693.79
Recpack	Correct exp	115.44	25.54 x	25.94 x	243.09	5681.55
Recpack	Correct full	115.35	25.52 x	25.92 x	243.27	5685.80

GRU4Rec_Tensorflow

Metric

BPR-Max

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Cross-Entropy

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.0647	0.0647	0.2220	0.1181	0.3414	0.1339	0.4874	0.1440
GRU4Rec Official	GRU4Rec_Tensorflow params	0.0313	0.0313	0.1136	0.0588	0.1893	0.0687	0.2976	0.0761
GRU4Rec_Tensorflow	OOB	0.0043	0.0043	0.0129	0.0073	0.0202	0.0082	0.0284	0.0088
GRU4Rec_Tensorflow	Correct Exp	0.0335	0.0335	0.1224	0.0635	0.1978	0.0734	0.3014	0.0805

Metric Diff

BPR-Max

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Cross-Entropy

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	GRU4Rec_Tensorflow params	-51.59%	-51.59%	-48.83%	-50.26%	-44.54%	-48.70%	-38.94%	-47.19%
GRU4Rec_Tensorflow	OOB	-93.40%	-93.40%	-94.19%	-93.83%	-94.08%	-93.85%	-94.17%	-93.89%
GRU4Rec_Tensorflow	Correct Exp	-48.27%	-48.27%	-44.85%	-46.24%	-42.07%	-45.18%	-38.16%	-44.14%

Hyperparameters

BPR-Max

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Cross-Entropy

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	GRU4Rec_Tensorflow	GRU4Rec_Tensorflow
Variant	Best params	GRU4Rec_Tensorflow params	OOB	Correct Exp
loss	cross-entropy	cross-entropy	cross-entropy	cross-entropy
optim	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False
embedding	0	192	192	192
final_act	softmax	softmax	softmax	softmax
layers	192	192	192	192
batch_size	128	128	50	128
dropout_p_embed	0.45	0	N/A	N/A
dropout_p_hidden	0.15	0.15	0.15	0.15
learning_rate	0.1	0.1	0.1	0.1
momentum	0	0	N/A	N/A
n_sample	2048	0	N/A	N/A
sample_alpha	0	0	N/A	N/A
bpreg	0	0	N/A	N/A
logq	1	0	N/A	N/A

Runtimes

BPR-Max

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Cross-Entropy

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	4.52			1134.52	144959.00
GRU4Rec Official	GRU4Rec_Tensorflow params	3.62	0.80 x		1417.96	181209.00
GRU4Rec_Tensorflow	OOB	26.33	5.83 x	7.27 x	498.25	24912.83
GRU4Rec_Tensorflow	Correct Exp	10.75	2.38 x	2.97 x	476.49	60990.52

KerasGRU4Rec

Metric

BPR-Max

Note

BPR-Max is not supported by KerasGRU4Rec

Cross-Entropy

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.0647	0.0647	0.2220	0.1181	0.3414	0.1339	0.4874	0.1440
GRU4Rec Official	KerasGRU4Rec params	0.0572	0.0572	0.1962	0.1042	0.3003	0.1178	0.4304	0.1268
GRU4Rec Official	KerasGRU4Rec params	0.0604	0.0604	0.2010	0.1086	0.3042	0.1223	0.4320	0.1311
KerasGRU4Rec	OOB	0.0511	0.0511	0.1723	0.0920	0.2704	0.1049	0.3922	0.1133
KerasGRU4Rec	Correct exp	0.0518	0.0518	0.1774	0.0945	0.2694	0.1066	0.3853	0.1146
KerasGRU4Rec	Correct full	0.0510	0.0510	0.1755	0.0935	0.2699	0.1060	0.3869	0.1141

Metric Diff

BPR-Max

Note

BPR-Max is not supported by KerasGRU4Rec

Cross-Entropy

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	KerasGRU4Rec params	-11.66%	-11.66%	-11.62%	-11.83%	-12.02%	-11.98%	-11.70%	-12.00%
GRU4Rec Official	KerasGRU4Rec params	-6.72%	-6.72%	-9.43%	-8.02%	-10.89%	-8.65%	-11.37%	-8.99%
KerasGRU4Rec	OOB	-21.10%	-21.10%	-22.38%	-22.09%	-20.80%	-21.61%	-19.53%	-21.32%
KerasGRU4Rec	Correct exp	-19.91%	-19.91%	-20.08%	-20.02%	-21.08%	-20.36%	-20.94%	-20.44%
KerasGRU4Rec	Correct full	-21.27%	-21.27%	-20.94%	-20.82%	-20.92%	-20.81%	-20.61%	-20.81%

Hyperparameters

BPR-Max

Note

BPR-Max is not supported by KerasGRU4Rec

Cross-Entropy

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	GRU4Rec Official	KerasGRU4Rec	KerasGRU4Rec	KerasGRU4Rec
Variant	Best params	KerasGRU4Rec params	KerasGRU4Rec params	OOB	Correct exp	Correct full
loss	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy
optim	adagrad	adagrad	adagrad	adam	adagrad	adagrad
constrained_embedding	True	False	False	False	False	False
embedding	0	0	0	0	0	0
final_act	softmax	softmax	softmax	softmax	softmax	softmax
layers	192	192	192	192	192	192
batch_size	128	128	128	100	128	128
dropout_p_embed	0.45	0	0	N/A	N/A	N/A
dropout_p_hidden	0.15	0.15	0.15	0.25	0.15	0.15
learning_rate	0.1	0.1	0.1	0.001	0.1	0.1
momentum	0	0	0	N/A	N/A	N/A
n_sample	2048	2048	ALL	N/A	N/A	N/A
sample_alpha	0	0	N/A	N/A	N/A	N/A
bpreg	0	0	0	N/A	N/A	N/A
logq	1	0	N/A	N/A	N/A	N/A

Runtimes

BPR-Max

Note

BPR-Max is not supported by KerasGRU4Rec

Cross-Entropy

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	4.52			1134.52	144959.00
GRU4Rec Official	KerasGRU4Rec params	4.06	0.90 x		1264.57	161575.00
GRU4Rec Official	KerasGRU4Rec params	74.84	16.56 x	18.43 x	68.58	8763.00
KerasGRU4Rec	OOB	359.31	79.49 x	88.50 x	14.26	1824.70
KerasGRU4Rec	Correct exp	355.51	78.65 x	87.56 x	14.45	1850.14
KerasGRU4Rec	Correct full	367.03	81.20 x	90.40 x	13.96	1786.86