bAbi

I thought it would be clever to solve bAbi, with the intention of creating my own dataset.

So with a deep interest in the subject in general and why my code was getting answers wrong in particular, I present a few minor nits to pick with the dataset (and a big one with task 2).

While most of these issues are rare, because this is synthetic data, errors are more serious, as they suggest problems with the underlying model.

Version of bAbi used: 1.2.

The authors:
Others:

Accuracy
1. 100.00% single supporting fact
2. 77.09% ? two supporting facts
3. 99.40% ? three supporting facts
4. 100.00% two arg relations
5. 99.80% ? three arg relations
6. 100.00% yes no questions
7. 100.00% counting
8. 99.60% ? lists sets
9. 100.00% simple negation
10. 100.00% indefinite knowledge
11. 100.00% basic coreference
12. 100.00% conjunction
13. 100.00% compound coreference
14. 100.00% time reasoning
15. 100.00% basic deduction
16. 97.30% ? basic induction
17. 100.00% positional reasoning
18. 100.00% size reasoning
19. 100.00% path finding
20. 100.00% agents motivations

Lots of coreference errors. I haven't checked each one, but I think about a quarter of the test answers are wrong in this manner.

1 Mary journeyed to the bathroom.
2 Sandra went to the garden.
3 Daniel went back to the garden.
4 Daniel went to the office.
5 Sandra grabbed the milk there.
6 Sandra put down the milk there.
7 Where is the milk? garden 6 2

Multiple correct answers.

1 John moved to the bathroom.
2 Sandra took the apple there.
3 Daniel journeyed to the hallway.
4 John journeyed to the hallway.
5 John went to the kitchen.
6 Sandra moved to the bedroom.
7 Sandra went back to the garden.
8 Sandra grabbed the football.
9 John travelled to the office.
10 John grabbed the milk.
11 Daniel went back to the office.
12 Sandra dropped the apple.
13 Where was the apple before the garden? bedroom 12 7 6
14 Mary moved to the garden.
15 Mary got the apple.
16 Mary moved to the office.
17 Daniel went back to the bedroom.
18 Mary went back to the kitchen.
19 Mary went to the hallway.
20 Where was the apple before the hallway? kitchen 15 19 18
21 Mary went back to the garden.
22 Mary discarded the apple.
23 Where was the apple before the hallway? kitchen 22 19 18
24 Mary went to the bedroom.
25 Sandra got the apple.
26 John moved to the kitchen.
27 Daniel went to the hallway.
28 Mary journeyed to the bathroom.
29 John moved to the garden.
30 John travelled to the kitchen.
31 John moved to the office.
32 Mary journeyed to the garden.
33 Daniel went to the kitchen.
34 John left the milk there.
35 Sandra discarded the apple.
36 Where was the milk before the office? kitchen 34 31 30
37 Sandra moved to the hallway.
38 Mary went to the bedroom.
39 Where was the milk before the kitchen? garden 34 30 29

Multiple correct answers. Surely a human would do badly here.

1 Mary journeyed to the bathroom.
2 Mary moved to the hallway.
3 Mary went to the kitchen.
4 Bill went back to the bedroom.
5 Bill grabbed the apple there.
6 Fred went back to the garden.
7 Mary went to the garden.
8 Fred took the milk there.
9 Jeff moved to the hallway.
10 Bill dropped the apple there.
11 Fred handed the milk to Mary.
12 Mary handed the milk to Fred.
13 Who gave the milk? Mary 12
14 Fred went back to the bedroom.
15 Fred passed the milk to Bill.
16 Who did Fred give the milk to? Bill 15
17 Fred took the apple there.
18 Fred gave the apple to Bill.
19 Who gave the apple? Fred 18
20 Jeff went to the kitchen.
21 Bill dropped the milk.
22 What did Fred give to Bill? milk 15

Items not listed in time or alphabetical order, but some other way.

1 John went to the hallway.
2 Daniel travelled to the hallway.
3 Mary went to the bathroom.
4 Daniel got the football there.
5 What is Daniel carrying? football 4
6 Mary journeyed to the office.
7 Daniel grabbed the apple there.
8 What is Daniel carrying? football,apple 4 7
9 Mary journeyed to the kitchen.
10 Daniel went to the bedroom.
11 What is Daniel carrying? football,apple 4 7
12 Daniel dropped the football there.
13 Daniel grabbed the football there.
14 What is Daniel carrying? football,apple 4 12 13 7

Surely this is confusing.

1 Bernhard is a rhino.
2 Brian is a rhino.
3 Bernhard is white.
4 Brian is white.
5 Lily is a lion.
6 Lily is yellow.
7 Greg is a rhino.
8 Greg is green.
9 Julius is a rhino.
10 What color is Julius? green 9 7 8