Replicate Jussi Rintanen’s benchmark set #16

New Issue

patrick · 2017-11-09T15:54:57+01:00

patrick commented

2017-11-09 15:54:57 +01:00

Jussi Rintanen sent us his benchmark set.

I started making a list of benchmark instances out of this archive (up to IPC 2006 currently), which I need to finish.

I couldn’t recover the following details about Rintanen’s benchmark set:

When Rintanen had two formulations of the same domain and the corresponding instances (with some domains, there were grounded and nongrounded formulations, for instance), I selected STRIPS formulations and nonground ones over the other ones.
ipc-1998/logistics-round-1-strips has 35 instances, yet Rintanen uses only 30. I decided to pick the first 30 instances for that reason.
ipc-1998/mystery-prime-round-1-strips has 35 instances, yet Rintanen uses only 20. I decided to pick the first 20 instances for that reason.
ipc-1998/mystery-round-1-strips has 30 instances, yet Rintanen uses only 19. I decided to pick the first 19 instances for that reason.
In ipc-2002, it appears that Rintanen used the automatic variations and not he hand-coded ones.
Concerning the IPC 2011 instance sets, it’s most likely that all instances were taken from the sequential, satisficing track, which is why I chose those instances.
Some domains and instances use action costs/metrics, which aren’t currently supported by the encodings. For this reason, the following domains were excluded from the benchmark:
- ipc-2008/cyber-security-sequential-satisficing-strips
- ipc-2011/barman-sequential-satisficing
- ipc-2011/elevator-sequential-satisficing
- ipc-2011/no-mystery-sequential-satisficing
- ipc-2011/openstacks-sequential-satisficing
- ipc-2011/parc-printer-sequential-satisficing
- ipc-2011/parking-sequential-satisficing
- ipc-2011/peg-solitaire-sequential-satisficing
- ipc-2011/scanalyzer-3d-sequential-satisficing
- ipc-2011/sokoban-sequential-satisficing
- ipc-2011/transport-sequential-satisficing
- ipc-2011/woodworking-sequential-satisficing

Jussi Rintanen sent us [his benchmark set](https://cloud.luehne.de/s/TuVplTvkzNG8jTO). I started making a list of benchmark instances out of this archive (up to IPC 2006 currently), which I need to finish. I couldn’t recover the following details about Rintanen’s benchmark set: - When Rintanen had two formulations of the same domain and the corresponding instances (with some domains, there were grounded and nongrounded formulations, for instance), I selected STRIPS formulations and nonground ones over the other ones. - `ipc-1998/logistics-round-1-strips` has 35 instances, yet Rintanen uses only 30. I decided to pick the first 30 instances for that reason. - `ipc-1998/mystery-prime-round-1-strips` has 35 instances, yet Rintanen uses only 20. I decided to pick the first 20 instances for that reason. - `ipc-1998/mystery-round-1-strips` has 30 instances, yet Rintanen uses only 19. I decided to pick the first 19 instances for that reason. - In `ipc-2002`, it appears that Rintanen used the `automatic` variations and not he `hand-coded` ones. - Concerning the IPC 2011 instance sets, it’s most likely that all instances were taken from the *sequential, satisficing* track, which is why I chose those instances. - Some domains and instances use action costs/metrics, which aren’t currently supported by the encodings. For this reason, the following domains were excluded from the benchmark: - `ipc-2008/cyber-security-sequential-satisficing-strips` - `ipc-2011/barman-sequential-satisficing` - `ipc-2011/elevator-sequential-satisficing` - `ipc-2011/no-mystery-sequential-satisficing` - `ipc-2011/openstacks-sequential-satisficing` - `ipc-2011/parc-printer-sequential-satisficing` - `ipc-2011/parking-sequential-satisficing` - `ipc-2011/peg-solitaire-sequential-satisficing` - `ipc-2011/scanalyzer-3d-sequential-satisficing` - `ipc-2011/sokoban-sequential-satisficing` - `ipc-2011/transport-sequential-satisficing` - `ipc-2011/woodworking-sequential-satisficing`

patrick added this to the Extended Paper for TPLP Journal milestone 2017-11-09 15:54:57 +01:00

patrick self-assigned this 2017-11-09 15:54:57 +01:00

patrick added the task label 2017-11-09 15:54:57 +01:00

patrick added the benchmarks label 2017-11-18 20:23:11 +01:00

patrick closed this issue

2017-12-07 13:55:35 +01:00

This repo is archived. You cannot comment on issues.

1 Participants

Due Date

-

Dependencies

No dependencies set.

Reference: patrick/plasp#16