Recpack

• Out-of-the-box:

  Running the code on GPU required fixes in the evaluation pipeline and the application of dropout.

• Minor fix:

  1. bpreg is no longer hard-coded.

  2. Fixed BPR-max so log() can no longer have 0 as its argument.

  3. Switched weight matrix initialization to Glorot uniform to match the original.

• Major fix:

  Fixed dropout to use separate parameters for embeddings and hidden states.

• Sampling is performed after all item scores are computed, which slows down training. This bug is rooted so deep in the code that we did not fix it.

Rees46

Metric

BPR-Max

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.1027	0.1027	0.2897	0.1680	0.4027	0.1831	0.5206	0.1913
GRU4Rec Official	Recpack params	0.0896	0.0896	0.2654	0.1508	0.3787	0.1658	0.5010	0.1743
Recpack	OOB	0.0762	0.0762	0.2346	0.1311	0.3405	0.1451	0.4624	0.1536
Recpack	Correct exp	0.0767	0.0767	0.2364	0.1321	0.3417	0.1461	0.4624	0.1544
Recpack	Correct full	0.0774	0.0774	0.2399	0.1338	0.3458	0.1478	0.4666	0.1562

Cross-Entropy

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.1108	0.1108	0.3000	0.1772	0.4126	0.1922	0.5291	0.2003
GRU4Rec Official	Recpack params	0.0840	0.0840	0.2531	0.1427	0.3625	0.1572	0.4812	0.1655
GRU4Rec Official	Recpack params	0.1078	0.1078	0.2937	0.1729	0.4034	0.1875	0.5174	0.1955
Recpack	OOB	0.0740	0.0740	0.2292	0.1281	0.3281	0.1412	0.4412	0.1490
Recpack	Correct exp	0.0754	0.0754	0.2304	0.1294	0.3288	0.1424	0.4414	0.1502
Recpack	Correct full	0.0754	0.0754	0.2299	0.1289	0.3294	0.1421	0.4386	0.1496

Metric Diff

BPR-Max

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	Recpack params	-12.73%	-12.73%	-8.41%	-10.27%	-5.98%	-9.42%	-3.76%	-8.85%
Recpack	OOB	-25.79%	-25.79%	-19.03%	-21.99%	-15.45%	-20.73%	-11.17%	-19.71%
Recpack	Correct exp	-25.30%	-25.30%	-18.41%	-21.37%	-15.16%	-20.21%	-11.17%	-19.25%
Recpack	Correct full	-24.63%	-24.63%	-17.21%	-20.38%	-14.15%	-19.26%	-10.37%	-18.34%

Cross-Entropy

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	Recpack params	-24.21%	-24.21%	-15.63%	-19.47%	-12.16%	-18.19%	-9.05%	-17.39%
GRU4Rec Official	Recpack params	-2.76%	-2.76%	-2.10%	-2.40%	-2.23%	-2.42%	-2.22%	-2.41%
Recpack	OOB	-33.20%	-33.20%	-23.61%	-27.73%	-20.48%	-26.53%	-16.61%	-25.60%
Recpack	Correct exp	-31.97%	-31.97%	-23.19%	-26.98%	-20.31%	-25.89%	-16.59%	-25.00%
Recpack	Correct full	-31.99%	-31.99%	-23.36%	-27.27%	-20.18%	-26.07%	-17.11%	-25.29%

Hyperparameters

BPR-Max

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	Recpack	Recpack	Recpack
Variant	Best params	Recpack params	OOB	Correct exp	Correct full
loss	bpr-max	bpr-max	bpr-max	bpr-max	bpr-max
optim	adagrad	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False	False
embedding	0	512	512	512	512
final_act	elu-0.5	linear	N/A	N/A	N/A
layers	512	512	512	512	512
batch_size	32	32	32	32	32
dropout_p_embed	0.1	0.1	0.1	0.1	0.1
dropout_p_hidden	0	0	0.1	0.1	0
learning_rate	0.03	0.03	0.03	0.03	0.03
momentum	0.55	0	N/A	N/A	N/A
n_sample	2048	2048	2048	2048	2048
sample_alpha	0.2	0	N/A	N/A	N/A
bpreg	0.75	0.75	0.75	0.75	0.75
logq	0	0	N/A	N/A	N/A

Cross-Entropy

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	GRU4Rec Official	Recpack	Recpack	Recpack
Variant	Best params	Recpack params	Recpack params	OOB	Correct exp	Correct full
loss	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy
optim	adagrad	adagrad	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False	False	False
embedding	0	512	512	512	512	512
final_act	softmax	softmax	softmax	softmax	softmax	softmax
layers	512	512	512	512	512	512
batch_size	240	240	240	240	240	240
dropout_p_embed	0.45	0.45	0.45	0.45	0.45	0.45
dropout_p_hidden	0	0	0	0.45	0.45	0
learning_rate	0.065	0.065	0.065	0.065	0.065	0.065
momentum	0	0	0	N/A	N/A	N/A
n_sample	2048	2048	ALL	N/A	N/A	N/A
sample_alpha	0.5	0	N/A	N/A	N/A	N/A
bpreg	0	0	0	0	0	0
logq	1	0	N/A	N/A	N/A	N/A

Runtimes

BPR-Max

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	1956.80			916.45	29326.00
GRU4Rec Official	Recpack params	1786.73	0.91 x		1003.68	32118.00
Recpack	OOB	179834.54	91.90 x	100.65 x	55.21	319.10
Recpack	Correct exp	179486.17	91.72 x	100.46 x	55.32	319.72
Recpack	Correct full	180173.69	92.08 x	100.84 x	55.10	318.50

Cross-Entropy

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	367.41			650.91	156189.00
GRU4Rec Official	Recpack params	382.60	1.04 x		625.07	149988.00
GRU4Rec Official	Recpack params	36064.38	98.16 x	94.26 x	6.63	1591.00
Recpack	OOB	49262.16	134.08 x	128.76 x	55.91	1164.89
Recpack	Correct exp	49366.03	134.36 x	129.03 x	55.79	1162.44
Recpack	Correct full	49253.13	134.05 x	128.73 x	55.92	1165.11

Yoochoose

Metric

BPR-Max

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.1745	0.1745	0.4346	0.2675	0.5664	0.2851	0.6799	0.2931
GRU4Rec Official	Recpack params	0.1753	0.1753	0.4304	0.2666	0.5578	0.2837	0.6749	0.2919
Recpack	OOB	0.1500	0.1500	0.3878	0.2358	0.5182	0.2532	0.6430	0.2619
Recpack	Correct exp	0.1571	0.1571	0.3984	0.2443	0.5260	0.2613	0.6464	0.2697
Recpack	Correct full	0.1581	0.1581	0.3967	0.2437	0.5239	0.2607	0.6433	0.2691

Cross-Entropy

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.1797	0.1797	0.4457	0.2757	0.5698	0.2924	0.6804	0.3002
GRU4Rec Official	Recpack params	0.1709	0.1709	0.4301	0.2625	0.5603	0.2800	0.6713	0.2877
GRU4Rec Official	Recpack params	0.1825	0.1825	0.4447	0.2770	0.5698	0.2938	0.6802	0.3016
Recpack	OOB	0.1533	0.1533	0.3941	0.2393	0.5108	0.2549	0.6214	0.2627
Recpack	Correct exp	0.1548	0.1548	0.3970	0.2417	0.5157	0.2576	0.6256	0.2653
Recpack	Correct full	0.1647	0.1647	0.4074	0.2516	0.5283	0.2678	0.6382	0.2755

Metric Diff

BPR-Max

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	Recpack params	0.43%	0.43%	-0.96%	-0.32%	-1.51%	-0.50%	-0.74%	-0.41%
Recpack	OOB	-14.04%	-14.04%	-10.77%	-11.85%	-8.51%	-11.20%	-5.44%	-10.63%
Recpack	Correct exp	-9.98%	-9.98%	-8.32%	-8.68%	-7.13%	-8.35%	-4.93%	-7.97%
Recpack	Correct full	-9.43%	-9.43%	-8.71%	-8.88%	-7.50%	-8.56%	-5.39%	-8.19%

Cross-Entropy

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	Recpack params	-4.89%	-4.89%	-3.52%	-4.81%	-1.67%	-4.25%	-1.33%	-4.14%
GRU4Rec Official	Recpack params	1.56%	1.56%	-0.23%	0.46%	0.00%	0.50%	-0.02%	0.46%
Recpack	OOB	-14.70%	-14.70%	-11.57%	-13.20%	-10.35%	-12.81%	-8.67%	-12.50%
Recpack	Correct exp	-13.87%	-13.87%	-10.94%	-12.35%	-9.50%	-11.89%	-8.05%	-11.62%
Recpack	Correct full	-8.33%	-8.33%	-8.59%	-8.77%	-7.29%	-8.41%	-6.19%	-8.23%

Hyperparameters

BPR-Max

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	Recpack	Recpack	Recpack
Variant	Best params	Recpack params	OOB	Correct exp	Correct full
loss	bpr-max	bpr-max	bpr-max	bpr-max	bpr-max
optim	adagrad	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False	False
embedding	0	448	448	448	448
final_act	linear	linear	N/A	N/A	N/A
layers	448	448	448	448	448
batch_size	48	48	48	48	48
dropout_p_embed	0.25	0.25	0.25	0.25	0.25
dropout_p_hidden	0	0	0.25	0.25	0
learning_rate	0.075	0.075	0.075	0.075	0.075
momentum	0.1	0	N/A	N/A	N/A
n_sample	2048	2048	2048	2048	2048
sample_alpha	0.2	0	N/A	N/A	N/A
bpreg	0.5	0.5	0.5	0.5	0.5
logq	0	0	N/A	N/A	N/A

Cross-Entropy

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	GRU4Rec Official	Recpack	Recpack	Recpack
Variant	Best params	Recpack params	Recpack params	OOB	Correct exp	Correct full
loss	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy
optim	adagrad	adagrad	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False	False	False
embedding	0	480	480	480	480	480
final_act	softmax	softmax	softmax	softmax	softmax	softmax
layers	480	480	480	480	480	480
batch_size	48	48	48	48	48	48
dropout_p_embed	0	0	0	0	0	0
dropout_p_hidden	0.2	0.2	0.2	0	0	0.2
learning_rate	0.07	0.07	0.07	0.07	0.07	0.07
momentum	0	0	0	N/A	N/A	N/A
n_sample	2048	2048	ALL	N/A	N/A	N/A
sample_alpha	0.2	0	N/A	N/A	N/A	N/A
bpreg	0	0	0	0	0	0
logq	1	0	N/A	N/A	N/A	N/A

Runtimes

BPR-Max

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	487.51			919.23	44121.00
GRU4Rec Official	Recpack params	460.73	0.95 x		972.67	46686.00
Recpack	OOB	21269.54	43.63 x	46.16 x	131.16	1011.29
Recpack	Correct exp	21382.12	43.86 x	46.41 x	130.47	1005.96
Recpack	Correct full	21137.61	43.36 x	45.88 x	131.98	1017.60

Cross-Entropy

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	451.75			991.99	47613.00
GRU4Rec Official	Recpack params	442.76	0.98 x		1012.15	48581.00
GRU4Rec Official	Recpack params	2859.14	6.33 x	6.46 x	156.74	7523.00
Recpack	OOB	13458.18	29.79 x	30.40 x	207.29	1598.25
Recpack	Correct exp	13462.56	29.80 x	30.41 x	207.22	1597.74
Recpack	Correct full	13693.73	30.31 x	30.93 x	203.72	1570.76

Coveo

Metric

BPR-Max

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.0501	0.0501	0.1464	0.0835	0.2172	0.0928	0.3123	0.0994
GRU4Rec Official	Recpack params	0.0471	0.0471	0.1418	0.0800	0.2104	0.0890	0.3003	0.0951
Recpack	OOB	0.0287	0.0287	0.0905	0.0499	0.1405	0.0565	0.2091	0.0612
Recpack	Correct exp	0.0265	0.0265	0.0855	0.0469	0.1309	0.0528	0.1953	0.0573
Recpack	Correct full	0.0205	0.0205	0.0643	0.0351	0.1005	0.0399	0.1536	0.0435

Cross-Entropy

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.0489	0.0489	0.1418	0.0814	0.2085	0.0901	0.2947	0.0960
GRU4Rec Official	Recpack params	0.0458	0.0458	0.1359	0.0770	0.2009	0.0856	0.2829	0.0912
GRU4Rec Official	Recpack params	0.0483	0.0483	0.1368	0.0788	0.2012	0.0873	0.2851	0.0931
Recpack	OOB	0.0381	0.0381	0.1108	0.0635	0.1635	0.0704	0.2335	0.0752
Recpack	Correct exp	0.0375	0.0375	0.1094	0.0622	0.1605	0.0689	0.2323	0.0739
Recpack	Correct full	0.0365	0.0365	0.1061	0.0603	0.1596	0.0673	0.2309	0.0722

Metric Diff

BPR-Max

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	Recpack params	-6.02%	-6.02%	-3.08%	-4.21%	-3.16%	-4.15%	-3.84%	-4.25%
Recpack	OOB	-42.83%	-42.83%	-38.18%	-40.25%	-35.29%	-39.19%	-33.04%	-38.46%
Recpack	Correct exp	-47.19%	-47.19%	-41.56%	-43.86%	-39.75%	-43.08%	-37.45%	-42.37%
Recpack	Correct full	-59.09%	-59.09%	-56.05%	-57.94%	-53.73%	-57.07%	-50.80%	-56.25%

Cross-Entropy

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	Recpack params	-6.26%	-6.26%	-4.13%	-5.39%	-3.64%	-5.06%	-4.01%	-5.02%
GRU4Rec Official	Recpack params	-1.28%	-1.28%	-3.50%	-3.08%	-3.53%	-3.14%	-3.27%	-3.07%
Recpack	OOB	-22.07%	-22.07%	-21.87%	-22.00%	-21.61%	-21.88%	-20.77%	-21.64%
Recpack	Correct exp	-23.40%	-23.40%	-22.85%	-23.52%	-23.04%	-23.50%	-21.17%	-23.06%
Recpack	Correct full	-25.41%	-25.41%	-25.15%	-25.92%	-23.45%	-25.33%	-21.65%	-24.80%

Hyperparameters

BPR-Max

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	Recpack	Recpack	Recpack
Variant	Best params	Recpack params	OOB	Correct exp	Correct full
loss	bpr-max	bpr-max	bpr-max	bpr-max	bpr-max
optim	adagrad	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False	False
embedding	0	512	512	512	512
final_act	elu-1	linear	N/A	N/A	N/A
layers	512	512	512	512	512
batch_size	144	144	144	144	144
dropout_p_embed	0.35	0.35	0.35	0.35	0.35
dropout_p_hidden	0	0	0.35	0.35	0
learning_rate	0.05	0.05	0.05	0.05	0.05
momentum	0.4	0	N/A	N/A	N/A
n_sample	2048	2048	2048	2048	2048
sample_alpha	0.2	0	N/A	N/A	N/A
bpreg	1.85	1.85	1	1.85	1.85
logq	0	0	N/A	N/A	N/A

Cross-Entropy

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	GRU4Rec Official	Recpack	Recpack	Recpack
Variant	Best params	Recpack params	Recpack params	OOB	Correct exp	Correct full
loss	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy
optim	adagrad	adagrad	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False	False	False
embedding	0	512	512	512	512	512
final_act	softmax	softmax	softmax	softmax	softmax	softmax
layers	512	512	512	512	512	512
batch_size	32	32	32	32	32	32
dropout_p_embed	0.4	0.4	0.4	0.4	0.4	0.4
dropout_p_hidden	0.15	0.15	0.15	0.4	0.4	0.15
learning_rate	0.03	0.03	0.03	0.03	0.03	0.03
momentum	0	0	N/A	N/A	N/A	N/A
n_sample	2048	2048	ALL	N/A	N/A	N/A
sample_alpha	0	0	N/A	N/A	N/A	N/A
bpreg	0	0	0	0	0	0
logq	1	0	N/A	N/A	N/A	N/A

Runtimes

BPR-Max

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	12.38			704.14	100615.00
GRU4Rec Official	Recpack params	12.19	0.98 x		715.00	102167.00
Recpack	OOB	770.91	62.27 x	63.24 x	110.48	1615.97
Recpack	Correct exp	760.89	61.46 x	62.42 x	111.91	1636.88
Recpack	Correct full	758.63	61.28 x	62.23 x	112.25	1641.79

Cross-Entropy

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	37.17			1047.47	33505.00
GRU4Rec Official	Recpack params	36.65	0.99 x		1062.34	33980.00
GRU4Rec Official	Recpack params	74.28	2.00 x	2.03 x	524.23	16768.00
Recpack	OOB	662.69	17.83 x	18.08 x	346.18	1881.49
Recpack	Correct exp	656.08	17.65 x	17.90 x	349.57	1899.95
Recpack	Correct full	637.93	17.16 x	17.41 x	359.21	1952.35

Retailrocket

Metric

BPR-Max

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.1224	0.1224	0.3196	0.1928	0.4181	0.2060	0.5187	0.2131
GRU4Rec Official	Recpack params	0.0883	0.0883	0.2491	0.1449	0.3449	0.1577	0.4455	0.1647
Recpack	OOB	0.0618	0.0618	0.1757	0.1010	0.2461	0.1104	0.3226	0.1157
Recpack	Correct exp	0.0567	0.0567	0.1667	0.0954	0.2320	0.1041	0.3054	0.1092
Recpack	Correct full	0.0641	0.0641	0.1902	0.1082	0.2679	0.1185	0.3550	0.1245

Cross-Entropy

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.1150	0.1150	0.3026	0.1814	0.4019	0.1947	0.4953	0.2012
GRU4Rec Official	Recpack params	0.0914	0.0914	0.2458	0.1457	0.3356	0.1577	0.4235	0.1637
GRU4Rec Official	Recpack params	0.1040	0.1040	0.2670	0.1620	0.3508	0.1732	0.4322	0.1788
Recpack	OOB	0.0609	0.0609	0.1514	0.0935	0.1960	0.0995	0.2442	0.1029
Recpack	Correct exp	0.0545	0.0545	0.1390	0.0845	0.1780	0.0897	0.2210	0.0927
Recpack	Correct full	0.0650	0.0650	0.1755	0.1044	0.2323	0.1119	0.2866	0.1157

Metric Diff

BPR-Max

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	Recpack params	-27.88%	-27.88%	-22.07%	-24.82%	-17.51%	-23.47%	-14.11%	-22.71%
Recpack	OOB	-49.51%	-49.51%	-45.04%	-47.59%	-41.14%	-46.42%	-37.80%	-45.71%
Recpack	Correct exp	-53.65%	-53.65%	-47.84%	-50.53%	-44.51%	-49.50%	-41.12%	-48.76%
Recpack	Correct full	-47.60%	-47.60%	-40.50%	-43.85%	-35.92%	-42.49%	-31.56%	-41.58%

Cross-Entropy

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	Recpack params	-20.58%	-20.58%	-18.79%	-19.70%	-16.48%	-19.03%	-14.48%	-18.64%
GRU4Rec Official	Recpack params	-9.63%	-9.63%	-11.77%	-10.72%	-12.71%	-11.05%	-12.74%	-11.14%
Recpack	OOB	-47.01%	-47.01%	-49.97%	-48.46%	-51.22%	-48.90%	-50.69%	-48.89%
Recpack	Correct exp	-52.61%	-52.61%	-54.07%	-53.44%	-55.70%	-53.94%	-55.38%	-53.95%
Recpack	Correct full	-43.53%	-43.53%	-41.99%	-42.48%	-42.20%	-42.51%	-42.14%	-42.51%

Hyperparameters

BPR-Max

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	Recpack	Recpack	Recpack
Variant	Best params	Recpack params	OOB	Correct exp	Correct full
loss	bpr-max	bpr-max	bpr-max	bpr-max	bpr-max
optim	adagrad	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False	False
embedding	0	224	224	224	224
final_act	elu-0.5	linear	N/A	N/A	N/A
layers	224	224	224	224	224
batch_size	80	80	80	80	80
dropout_p_embed	0.5	0.5	0.5	0.5	0.5
dropout_p_hidden	0.05	0.05	0.5	0.5	0.05
learning_rate	0.05	0.05	0.05	0.05	0.05
momentum	0.4
n_sample	2048	2048	2048	2048	2048
sample_alpha	0.4	0	N/A	N/A	N/A
bpreg	1.95	1.95	1	1.95	1.95
logq	0	0	N/A	N/A	N/A

Cross-Entropy

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	GRU4Rec Official	Recpack	Recpack	Recpack
Variant	Best params	Recpack params	Recpack params	OOB	Correct exp	Correct full
loss	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy
optim	adagrad	adagrad	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False	False	False
embedding	0	192	192	192	192	192
final_act	softmax	softmax	softmax	softmax	softmax	softmax
layers	192	192	192	192	192	192
batch_size	240	240	240	240	240	240
dropout_p_embed	0.5	0.5	0.5	0.5	0.5	0.5
dropout_p_hidden	0.05	0.05	0.05	0.5	0.5	0.05
learning_rate	0.085	0.085	0.085	0.085	0.085	0.085
momentum	0.3	0	0	N/A	N/A	N/A
n_sample	2048	2048	ALL	N/A	N/A	N/A
sample_alpha	0.3	0	N/A	N/A	N/A	N/A
bpreg	0	0	0	0	0	0
logq	1	0	N/A	N/A	N/A	N/A

Runtimes

BPR-Max

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	6.86			1019.34	80807.00
GRU4Rec Official	Recpack params	6.43	0.94 x		1087.28	86193.00
Recpack	OOB	450.18	65.62 x	70.01 x	125.44	1232.31
Recpack	Correct exp	451.63	65.84 x	70.24 x	125.05	1228.42
Recpack	Correct full	461.24	67.24 x	71.73 x	122.40	1202.41

Cross-Entropy

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	2.77			880.71	199935.00
GRU4Rec Official	Recpack params	2.45	0.88 x		997.22	226383.00
GRU4Rec Official	Recpack params	72.09	26.03 x	29.42 x	33.89	7693.00
Recpack	OOB	122.27	44.14 x	49.91 x	253.53	4535.76
Recpack	Correct exp	126.13	45.53 x	51.48 x	245.85	4398.31
Recpack	Correct full	121.39	43.82 x	49.55 x	255.38	4568.79

Diginetica

Metric

BPR-Max

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.0688	0.0688	0.2304	0.1237	0.3533	0.1399	0.4995	0.1500
GRU4Rec Official	Recpack params	0.0635	0.0635	0.2101	0.1138	0.3203	0.1283	0.4586	0.1379
Recpack	OOB	0.0409	0.0409	0.1412	0.0752	0.2162	0.0851	0.3066	0.0913
Recpack	Correct exp	0.0449	0.0449	0.1609	0.0839	0.2558	0.0964	0.3744	0.1046
Recpack	Correct full	0.0430	0.0430	0.1643	0.0836	0.2580	0.0959	0.3756	0.1040

Cross-Entropy

Metrics

Implementation	Variant	Recall@1	MRR@1	Recall@5	MRR@5	Recall@10	MRR@10	Recall@20	MRR@20
GRU4Rec Official	Best params	0.0647	0.0647	0.2220	0.1181	0.3414	0.1339	0.4874	0.1440
GRU4Rec Official	Recpack params	0.0562	0.0562	0.1902	0.1019	0.2939	0.1157	0.4263	0.1248
GRU4Rec Official	Recpack params	0.0603	0.0603	0.2001	0.1083	0.3058	0.1223	0.4341	0.1312
Recpack	OOB	0.0411	0.0411	0.1442	0.0760	0.2238	0.0865	0.3279	0.0937
Recpack	Correct exp	0.0411	0.0411	0.1433	0.0757	0.2193	0.0857	0.3219	0.0927
Recpack	Correct full	0.0408	0.0408	0.1463	0.0766	0.2274	0.0873	0.3296	0.0944

Metric Diff

BPR-Max

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	Recpack params	-7.71%	-7.71%	-8.82%	-7.99%	-9.35%	-8.24%	-8.19%	-8.05%
Recpack	OOB	-40.57%	-40.57%	-38.70%	-39.21%	-38.82%	-39.16%	-38.63%	-39.10%
Recpack	Correct exp	-34.84%	-34.84%	-30.16%	-32.13%	-27.59%	-31.06%	-25.05%	-30.27%
Recpack	Correct full	-37.51%	-37.51%	-28.70%	-32.38%	-26.96%	-31.41%	-24.81%	-30.62%

Cross-Entropy

Metric difference compared to the “Best params” version with the corresponding loss

Implementation	Variant	Recall@1 Diff	MRR@1 Diff	Recall@5 Diff	MRR@5 Diff	Recall@10 Diff	MRR@10 Diff	Recall@20 Diff	MRR@20 Diff
GRU4Rec Official	Best params
GRU4Rec Official	Recpack params	-13.23%	-13.23%	-14.32%	-13.69%	-13.90%	-13.59%	-12.52%	-13.34%
GRU4Rec Official	Recpack params	-6.88%	-6.88%	-9.87%	-8.32%	-10.41%	-8.62%	-10.92%	-8.93%
Recpack	OOB	-36.53%	-36.53%	-35.05%	-35.64%	-34.45%	-35.37%	-32.72%	-34.94%
Recpack	Correct exp	-36.50%	-36.50%	-35.44%	-35.94%	-35.75%	-36.00%	-33.96%	-35.62%
Recpack	Correct full	-37.00%	-37.00%	-34.07%	-35.11%	-33.38%	-34.75%	-32.37%	-34.46%

Hyperparameters

BPR-Max

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	Recpack	Recpack	Recpack
Variant	Best params	Recpack params	OOB	Correct exp	Correct full
loss	bpr-max	bpr-max	bpr-max	bpr-max	bpr-max
optim	adagrad	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False	False
embedding	0	512	512	512	512
final_act	elu-1	linear	N/A	N/A	N/A
layers	512	512	512	512	512
batch_size	128	128	128	128	128
dropout_p_embed	0.5	0.5	0.5	0.5	0.5
dropout_p_hidden	0.3	0.3	0.5	0.5	0.3
learning_rate	0.05	0.05	0.05	0.05	0.05
momentum	0.15	0	N/A	N/A	N/A
n_sample	2048	2048	2048	2048	2048
sample_alpha	0.3	0	N/A	N/A	N/A
bpreg	0.9	0.9	1	0.9	0.9
logq	0	0	N/A	N/A	N/A

Cross-Entropy

Hyperparameters used in the experiment

	GRU4Rec Official	GRU4Rec Official	GRU4Rec Official	Recpack	Recpack	Recpack
Variant	Best params	Recpack params	Recpack params	OOB	Correct exp	Correct full
loss	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy	cross-entropy
optim	adagrad	adagrad	adagrad	adagrad	adagrad	adagrad
constrained_embedding	True	False	False	False	False	False
embedding	0	192	192	192	192	192
final_act	softmax	softmax	softmax	softmax	softmax	softmax
layers	192	192	192	192	192	192
batch_size	128	128	128	128	128	128
dropout_p_embed	0.45	0.45	0.45	0.45	0.45	0.45
dropout_p_hidden	0.15	0.15	0.15	0.45	0.45	0.15
learning_rate	0.1	0.1	0.1	0.1	0.1	0.1
momentum	0	0	0	N/A	N/A	N/A
n_sample	2048	2048	ALL	N/A	N/A	N/A
sample_alpha	0	0	N/A	N/A	N/A	N/A
bpreg	0	0	0	0	0	0
logq	1	0	N/A	N/A	N/A	N/A

Runtimes

BPR-Max

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	8.02			639.87	81757.00
GRU4Rec Official	Recpack params	7.73	0.96 x		663.79	84812.00
Recpack	OOB	352.17	43.91 x	45.56 x	79.68	1862.33
Recpack	Correct exp	346.70	43.23 x	44.85 x	80.94	1891.72
Recpack	Correct full	342.34	42.69 x	44.29 x	81.97	1915.79

Cross-Entropy

Runtime metrics

Implementation	Variant	Avg. epoch time (s)	Avg. epoch time to Best	Avg. epoch time to Matching	Avg. mb/s	Avg. e/s
GRU4Rec Official	Best params	4.52			1134.52	144959.00
GRU4Rec Official	Recpack params	4.45	0.98 x		1154.05	147454.00
GRU4Rec Official	Recpack params	76.19	16.86 x	17.12 x	67.37	8608.00
Recpack	OOB	115.21	25.49 x	25.89 x	243.62	5693.79
Recpack	Correct exp	115.44	25.54 x	25.94 x	243.09	5681.55
Recpack	Correct full	115.35	25.52 x	25.92 x	243.27	5685.80