Repository navigation
Fix #205: run the program on every test id in problem 23 - #207
Closed
Kingpython123 wants to merge 1 commit into
Closed
Kingpython123 wants to merge 1 commit into
Kingpython123 wants to merge 1 commit into
Conversation
check_Tid() returned ok without running the submitted program when Tid % 100 == 0, so any parseable A=B program, including an empty one, was accepted on such tests. Test set 0 was empty, which is why the shortcut existed. Remove the shortcut and give set 0 real cases: the fixed samples plus the seeded random cases. Every test id now runs the program, and ids whose set was previously empty no longer score nan. Add algorithmic/tests/test_problem23.py as a regression test.
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
check_Tid()inalgorithmic/problems/23/check.cppcalledquitf(_ok, "Test 0.")whenTid % 100 == 0, before the submitted program was run. Any output that parses as an A=B program was therefore accepted on such tests, including an empty program and a single rule likea=b. At the commit pinned by Inspect Evals (c7ec558) all three test ids were 0, 100 and 200; on main none of the 16 ids is a multiple of 100, but the shortcut was still there.The shortcut existed because test set 0 was the empty generator, so running the program on it would have divided by zero. This PR removes the shortcut and gives set 0 real cases: the four fixed samples plus the 100 seeded random cases (the same generators as sets 1 and 2, seeded by
Tid / 100). Every test id now runs the program. As a side effect, test ids whoseTid % 100had no generator (21 to 99) also used to yield an empty set and a score ofnan; they now run set 0 as well.The 16 existing test ids are not affected: for them the generated cases and the scores are identical before and after (for example test 1, id 795913901, scores
Ratio: 0.613672131215with the program in1.anson both).Reproduction (g++ 11.4,
-O2 -std=gnu++17, repo testlib; same result with testlib 1e4e8a24):Before (main), for ids 0, 100 and 200:
After:
algorithmic/tests/test_problem23.pyis a regression test in the style of the existing checker tests: for ids 0, 100, 200 and 50 the empty and one-rule programs are WA and the program in1.ansscores, and an existing test id behaves as before. All 13 tests pass with this change; on main 12 of them fail (only the existing-id test passes).Fixes #205.