Skip to content

Fix #205: run the program on every test id in problem 23 - #207

Closed
Kingpython123 wants to merge 1 commit into
FrontierCS:mainfrom
Kingpython123:fix-205-problem23-tid-shortcut
Closed

Kingpython123 wants to merge 1 commit into
FrontierCS:mainfrom
Kingpython123:fix-205-problem23-tid-shortcut

Conversation

@Kingpython123

Copy link
Copy Markdown

check_Tid() in algorithmic/problems/23/check.cpp called quitf(_ok, "Test 0.") when Tid % 100 == 0, before the submitted program was run. Any output that parses as an A=B program was therefore accepted on such tests, including an empty program and a single rule like a=b. At the commit pinned by Inspect Evals (c7ec558) all three test ids were 0, 100 and 200; on main none of the 16 ids is a multiple of 100, but the shortcut was still there.

The shortcut existed because test set 0 was the empty generator, so running the program on it would have divided by zero. This PR removes the shortcut and gives set 0 real cases: the four fixed samples plus the 100 seeded random cases (the same generators as sets 1 and 2, seeded by Tid / 100). Every test id now runs the program. As a side effect, test ids whose Tid % 100 had no generator (21 to 99) also used to yield an empty set and a score of nan; they now run set 0 as well.

The 16 existing test ids are not affected: for them the generated cases and the scores are identical before and after (for example test 1, id 795913901, scores Ratio: 0.613672131215 with the program in 1.ans on both).

Reproduction (g++ 11.4, -O2 -std=gnu++17, repo testlib; same result with testlib 1e4e8a24):

g++ -O2 -std=gnu++17 -I algorithmic/judge/include -o chk algorithmic/problems/23/check.cpp
echo 0 > tid0.in; : > empty.out; printf 'a=b\n' > onerule.out
./chk tid0.in empty.out algorithmic/problems/23/testdata/1.ans
./chk tid0.in onerule.out algorithmic/problems/23/testdata/1.ans
./chk tid0.in algorithmic/problems/23/testdata/1.ans algorithmic/problems/23/testdata/1.ans

Before (main), for ids 0, 100 and 200:

ok Test 0.                      (exit 0)   empty program
ok Test 0.                      (exit 0)   a=b
ok Test 0.                      (exit 0)   program from 1.ans

After:

wrong answer WA on Test case 1. abcabbabcScabb 1   (exit 1)   empty program
wrong answer WA on Test case 1. abcabbabcScabb 1   (exit 1)   a=b
points 0.615449792 Ratio: 0.615449791957, ...     (exit 7)   program from 1.ans

algorithmic/tests/test_problem23.py is a regression test in the style of the existing checker tests: for ids 0, 100, 200 and 50 the empty and one-rule programs are WA and the program in 1.ans scores, and an existing test id behaves as before. All 13 tests pass with this change; on main 12 of them fail (only the existing-id test passes).

Fixes #205.

check_Tid() returned ok without running the submitted program when
Tid % 100 == 0, so any parseable A=B program, including an empty one,
was accepted on such tests. Test set 0 was empty, which is why the
shortcut existed. Remove the shortcut and give set 0 real cases: the
fixed samples plus the seeded random cases. Every test id now runs
the program, and ids whose set was previously empty no longer score nan.

Add algorithmic/tests/test_problem23.py as a regression test.
@Kingpython123

Copy link
Copy Markdown
Author

Thanks for the quick fix! This is superseded by #210, which landed the fix for #205, so I'm closing this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Problem 23 (A=B): checker returns ok without running the program when the test id is a multiple of 100

1 participant