GRU4Rec_Tensorflow#

  • Out-of-the-box:

    The code is able to run, however it is not prepared for unknown test items. The dropout parameter was fixed to regain its original meaning as it was used as keep probability instead of drop probability.

  • Minor fix:

    1. Lowered the large initial accumulator value close to zero as it prevented sufficient convergence.

    2. The exponential learning rate decay is discarded as it is not part of the original.

    3. Now the hard-coded batch_size parameter can be changed.

Rees46#

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Metrics#

Implementation

Variant

Recall@1

MRR@1

Recall@5

MRR@5

Recall@10

MRR@10

Recall@20

MRR@20

GRU4Rec Official

Best params

0.1108

0.1108

0.3000

0.1772

0.4126

0.1922

0.5291

0.2003

GRU4Rec Official

GRU4Rec_Tensorflow params

0.0304

0.0304

0.1178

0.0596

0.1950

0.0697

0.2940

0.0765

GRU4Rec_Tensorflow

OOB

0.0270

0.0270

0.0895

0.0482

0.1403

0.0549

0.2066

0.0594

GRU4Rec_Tensorflow

Correct Exp

0.0251

0.0251

0.1026

0.0507

0.1747

0.0602

0.2698

0.0667

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Metric difference compared to the “Best params” version with the corresponding loss#

Implementation

Variant

Recall@1 Diff

MRR@1 Diff

Recall@5 Diff

MRR@5 Diff

Recall@10 Diff

MRR@10 Diff

Recall@20 Diff

MRR@20 Diff

GRU4Rec Official

Best params

GRU4Rec Official

GRU4Rec_Tensorflow params

-72.60%

-72.60%

-60.75%

-66.39%

-52.75%

-63.75%

-44.43%

-61.81%

GRU4Rec_Tensorflow

OOB

-75.62%

-75.62%

-70.17%

-72.81%

-65.99%

-71.45%

-60.95%

-70.33%

GRU4Rec_Tensorflow

Correct Exp

-77.39%

-77.39%

-65.79%

-71.36%

-57.67%

-68.69%

-49.00%

-66.69%

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Hyperparameters used in the experiment#

GRU4Rec Official

GRU4Rec Official

GRU4Rec_Tensorflow

GRU4Rec_Tensorflow

Variant

Best params

GRU4Rec_Tensorflow params

OOB

Correct Exp

loss

cross-entropy

cross-entropy

cross-entropy

cross-entropy

optim

adagrad

adagrad

adagrad

adagrad

constrained_embedding

True

False

False

False

embedding

0

512

512

512

final_act

softmax

softmax

softmax

softmax

layers

512

512

512

512

batch_size

240

240

50

240

dropout_p_embed

0.45

0

N/A

N/A

dropout_p_hidden

0

0

0

0

learning_rate

0.065

0.065

0.065

0.065

momentum

0

0

N/A

N/A

n_sample

2048

0

N/A

N/A

sample_alpha

0.5

0

N/A

N/A

bpreg

0

0

N/A

N/A

logq

1

0

N/A

N/A

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Runtime metrics#

Implementation

Variant

Avg. epoch time (s)

Avg. epoch time to Best

Avg. epoch time to Matching

Avg. mb/s

Avg. e/s

GRU4Rec Official

Best params

367.41

650.91

156189.00

GRU4Rec Official

GRU4Rec_Tensorflow params

280.85

0.76 x

851.46

204324.00

GRU4Rec_Tensorflow

OOB

2381.62

6.48 x

8.48 x

481.90

24094.84

GRU4Rec_Tensorflow

Correct Exp

531.14

1.45 x

1.89 x

450.16

108037.24

Yoochoose#

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Metrics#

Implementation

Variant

Recall@1

MRR@1

Recall@5

MRR@5

Recall@10

MRR@10

Recall@20

MRR@20

GRU4Rec Official

Best params

0.1797

0.1797

0.4457

0.2757

0.5698

0.2924

0.6804

0.3002

GRU4Rec Official

GRU4Rec_Tensorflow params

0.0715

0.0715

0.2325

0.1269

0.3431

0.1416

0.4555

0.1495

GRU4Rec_Tensorflow

OOB

0.0727

0.0727

0.2051

0.1178

0.2946

0.1296

0.3857

0.1359

GRU4Rec_Tensorflow

Correct Exp

0.0691

0.0691

0.2301

0.1252

0.3409

0.1399

0.4606

0.1482

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Metric difference compared to the “Best params” version with the corresponding loss#

Implementation

Variant

Recall@1 Diff

MRR@1 Diff

Recall@5 Diff

MRR@5 Diff

Recall@10 Diff

MRR@10 Diff

Recall@20 Diff

MRR@20 Diff

GRU4Rec Official

Best params

GRU4Rec Official

GRU4Rec_Tensorflow params

-60.19%

-60.19%

-47.85%

-53.96%

-39.79%

-51.56%

-33.05%

-50.21%

GRU4Rec_Tensorflow

OOB

-59.52%

-59.52%

-54.00%

-57.27%

-48.30%

-55.68%

-43.31%

-54.73%

GRU4Rec_Tensorflow

Correct Exp

-61.54%

-61.54%

-48.37%

-54.61%

-40.17%

-52.16%

-32.29%

-50.63%

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Hyperparameters used in the experiment#

GRU4Rec Official

GRU4Rec Official

GRU4Rec_Tensorflow

GRU4Rec_Tensorflow

Variant

Best params

GRU4Rec_Tensorflow params

OOB

Correct Exp

loss

cross-entropy

cross-entropy

cross-entropy

cross-entropy

optim

adagrad

adagrad

adagrad

adagrad

constrained_embedding

True

False

False

False

embedding

0

480

480

480

final_act

softmax

softmax

softmax

softmax

layers

480

480

480

480

batch_size

48

48

50

48

dropout_p_embed

0

0

N/A

N/A

dropout_p_hidden

0.2

0.2

0.2

0.2

learning_rate

0.07

0.07

0.07

0.07

momentum

0

0

N/A

N/A

n_sample

2048

0

N/A

N/A

sample_alpha

0.2

0

N/A

N/A

bpreg

0

0

N/A

N/A

logq

1

0

N/A

N/A

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Runtime metrics#

Implementation

Variant

Avg. epoch time (s)

Avg. epoch time to Best

Avg. epoch time to Matching

Avg. mb/s

Avg. e/s

GRU4Rec Official

Best params

451.75

991.99

47613.00

GRU4Rec Official

GRU4Rec_Tensorflow params

352.48

0.78 x

1271.36

61023.00

GRU4Rec_Tensorflow

OOB

981.87

2.17 x

2.79 x

440.53

22026.42

GRU4Rec_Tensorflow

Correct Exp

965.98

2.14 x

2.74 x

463.92

22268.35

Coveo#

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Metrics#

Implementation

Variant

Recall@1

MRR@1

Recall@5

MRR@5

Recall@10

MRR@10

Recall@20

MRR@20

GRU4Rec Official

Best params

0.0489

0.0489

0.1418

0.0814

0.2085

0.0901

0.2947

0.0960

GRU4Rec Official

GRU4Rec_Tensorflow params

0.0273

0.0273

0.0853

0.0468

0.1326

0.0530

0.2001

0.0576

GRU4Rec_Tensorflow

OOB

0.0025

0.0025

0.0103

0.0050

0.0204

0.0063

0.0375

0.0075

GRU4Rec_Tensorflow

Correct Exp

0.0252

0.0252

0.0804

0.0441

0.1276

0.0503

0.1951

0.0549

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Metric difference compared to the “Best params” version with the corresponding loss#

Implementation

Variant

Recall@1 Diff

MRR@1 Diff

Recall@5 Diff

MRR@5 Diff

Recall@10 Diff

MRR@10 Diff

Recall@20 Diff

MRR@20 Diff

GRU4Rec Official

Best params

GRU4Rec Official

GRU4Rec_Tensorflow params

-44.10%

-44.10%

-39.85%

-42.52%

-36.42%

-41.24%

-32.12%

-40.06%

GRU4Rec_Tensorflow

OOB

-94.84%

-94.84%

-92.70%

-93.88%

-90.24%

-93.01%

-87.27%

-92.22%

GRU4Rec_Tensorflow

Correct Exp

-48.49%

-48.49%

-43.33%

-45.79%

-38.80%

-44.18%

-33.81%

-42.79%

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Hyperparameters used in the experiment#

GRU4Rec Official

GRU4Rec Official

GRU4Rec_Tensorflow

GRU4Rec_Tensorflow

Variant

Best params

GRU4Rec_Tensorflow params

OOB

Correct Exp

loss

cross-entropy

cross-entropy

cross-entropy

cross-entropy

optim

adagrad

adagrad

adagrad

adagrad

constrained_embedding

True

False

False

False

embedding

0

512

512

512

final_act

softmax

softmax

softmax

softmax

layers

512

512

512

512

batch_size

32

32

50

32

dropout_p_embed

0.4

0

N/A

N/A

dropout_p_hidden

0.15

0.15

0.15

0.15

learning_rate

0.03

0.03

0.03

0.03

momentum

0

0

N/A

N/A

n_sample

2048

0

N/A

N/A

sample_alpha

0

0

N/A

N/A

bpreg

0

0

N/A

N/A

logq

1

0

N/A

N/A

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Runtime metrics#

Implementation

Variant

Avg. epoch time (s)

Avg. epoch time to Best

Avg. epoch time to Matching

Avg. mb/s

Avg. e/s

GRU4Rec Official

Best params

37.17

1047.47

33505.00

GRU4Rec Official

GRU4Rec_Tensorflow params

29.40

0.79 x

1324.38

42363.00

GRU4Rec_Tensorflow

OOB

55.39

1.49 x

1.88 x

449.67

22483.60

GRU4Rec_Tensorflow

Correct Exp

81.90

2.20 x

2.79 x

475.29

15209.23

Retailrocket#

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Metrics#

Implementation

Variant

Recall@1

MRR@1

Recall@5

MRR@5

Recall@10

MRR@10

Recall@20

MRR@20

GRU4Rec Official

Best params

0.1150

0.1150

0.3026

0.1814

0.4019

0.1947

0.4953

0.2012

GRU4Rec Official

GRU4Rec_Tensorflow params

0.0541

0.0541

0.1662

0.0931

0.2425

0.1031

0.3309

0.1092

GRU4Rec_Tensorflow

OOB

0.0130

0.0130

0.0328

0.0201

0.0381

0.0208

0.0439

0.0212

GRU4Rec_Tensorflow

Correct Exp

0.0612

0.0612

0.1776

0.1019

0.2541

0.1120

0.3388

0.1179

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Metric difference compared to the “Best params” version with the corresponding loss#

Implementation

Variant

Recall@1 Diff

MRR@1 Diff

Recall@5 Diff

MRR@5 Diff

Recall@10 Diff

MRR@10 Diff

Recall@20 Diff

MRR@20 Diff

GRU4Rec Official

Best params

GRU4Rec Official

GRU4Rec_Tensorflow params

-52.99%

-52.99%

-45.08%

-48.71%

-39.67%

-47.05%

-33.19%

-45.73%

GRU4Rec_Tensorflow

OOB

-88.71%

-88.71%

-89.16%

-88.92%

-90.51%

-89.32%

-91.14%

-89.46%

GRU4Rec_Tensorflow

Correct Exp

-46.76%

-46.76%

-41.29%

-43.85%

-36.77%

-42.45%

-31.59%

-41.40%

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Hyperparameters used in the experiment#

GRU4Rec Official

GRU4Rec Official

GRU4Rec_Tensorflow

GRU4Rec_Tensorflow

Variant

Best params

GRU4Rec_Tensorflow params

OOB

Correct Exp

loss

cross-entropy

cross-entropy

cross-entropy

cross-entropy

optim

adagrad

adagrad

adagrad

adagrad

constrained_embedding

True

False

False

False

embedding

0

192

192

192

final_act

softmax

softmax

softmax

softmax

layers

192

192

192

192

batch_size

240

240

50

240

dropout_p_embed

0.5

0

N/A

N/A

dropout_p_hidden

0.05

0.05

0.05

0.05

learning_rate

0.085

0.085

0.085

0.085

momentum

0.3

0

N/A

N/A

n_sample

2048

0

N/A

N/A

sample_alpha

0.3

0

N/A

N/A

bpreg

0

0

N/A

N/A

logq

1

0

N/A

N/A

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Runtime metrics#

Implementation

Variant

Avg. epoch time (s)

Avg. epoch time to Best

Avg. epoch time to Matching

Avg. mb/s

Avg. e/s

GRU4Rec Official

Best params

2.77

880.71

199935.00

GRU4Rec Official

GRU4Rec_Tensorflow params

1.86

0.67 x

1298.62

297964.00

GRU4Rec_Tensorflow

OOB

22.97

8.29 x

12.35 x

482.83

24141.37

GRU4Rec_Tensorflow

Correct Exp

5.09

1.84 x

2.74 x

454.82

109157.42

Diginetica#

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Metrics#

Implementation

Variant

Recall@1

MRR@1

Recall@5

MRR@5

Recall@10

MRR@10

Recall@20

MRR@20

GRU4Rec Official

Best params

0.0647

0.0647

0.2220

0.1181

0.3414

0.1339

0.4874

0.1440

GRU4Rec Official

GRU4Rec_Tensorflow params

0.0313

0.0313

0.1136

0.0588

0.1893

0.0687

0.2976

0.0761

GRU4Rec_Tensorflow

OOB

0.0043

0.0043

0.0129

0.0073

0.0202

0.0082

0.0284

0.0088

GRU4Rec_Tensorflow

Correct Exp

0.0335

0.0335

0.1224

0.0635

0.1978

0.0734

0.3014

0.0805

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Metric difference compared to the “Best params” version with the corresponding loss#

Implementation

Variant

Recall@1 Diff

MRR@1 Diff

Recall@5 Diff

MRR@5 Diff

Recall@10 Diff

MRR@10 Diff

Recall@20 Diff

MRR@20 Diff

GRU4Rec Official

Best params

GRU4Rec Official

GRU4Rec_Tensorflow params

-51.59%

-51.59%

-48.83%

-50.26%

-44.54%

-48.70%

-38.94%

-47.19%

GRU4Rec_Tensorflow

OOB

-93.40%

-93.40%

-94.19%

-93.83%

-94.08%

-93.85%

-94.17%

-93.89%

GRU4Rec_Tensorflow

Correct Exp

-48.27%

-48.27%

-44.85%

-46.24%

-42.07%

-45.18%

-38.16%

-44.14%

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Hyperparameters used in the experiment#

GRU4Rec Official

GRU4Rec Official

GRU4Rec_Tensorflow

GRU4Rec_Tensorflow

Variant

Best params

GRU4Rec_Tensorflow params

OOB

Correct Exp

loss

cross-entropy

cross-entropy

cross-entropy

cross-entropy

optim

adagrad

adagrad

adagrad

adagrad

constrained_embedding

True

False

False

False

embedding

0

192

192

192

final_act

softmax

softmax

softmax

softmax

layers

192

192

192

192

batch_size

128

128

50

128

dropout_p_embed

0.45

0

N/A

N/A

dropout_p_hidden

0.15

0.15

0.15

0.15

learning_rate

0.1

0.1

0.1

0.1

momentum

0

0

N/A

N/A

n_sample

2048

0

N/A

N/A

sample_alpha

0

0

N/A

N/A

bpreg

0

0

N/A

N/A

logq

1

0

N/A

N/A

Note

BPR-Max is not supported by GRU4Rec_Tensorflow

Runtime metrics#

Implementation

Variant

Avg. epoch time (s)

Avg. epoch time to Best

Avg. epoch time to Matching

Avg. mb/s

Avg. e/s

GRU4Rec Official

Best params

4.52

1134.52

144959.00

GRU4Rec Official

GRU4Rec_Tensorflow params

3.62

0.80 x

1417.96

181209.00

GRU4Rec_Tensorflow

OOB

26.33

5.83 x

7.27 x

498.25

24912.83

GRU4Rec_Tensorflow

Correct Exp

10.75

2.38 x

2.97 x

476.49

60990.52