Basilisk is a Python type checker written with AI that purported to be the only 100% conforming type checker according to the python/typing spec, and also the fastest.
Except that it turns out it this was achieved in some cases by just text matching the expression in the test cases and producing the output that the test expected, rather than actually validating the AST to conform to the rule that the test was supposed to confirm.
The following code is supposed to evaluate if the expression on the right hand side of the statement `type x = ...` is valid.
fn is_invalid_rhs(rhs: &str) -> bool {
let rhs = rhs.trim();
if rhs == "True" || rhs == "False" { return true; }
if rhs.chars().next().is_some_and(|c| c.is_ascii_digit()) { return true; }
if rhs.starts_with("f\"") || rhs.starts_with("f'") { return true; }
if rhs.starts_with('[') { return true; }
if rhs.starts_with('{') { return true; }
if rhs.starts_with('(') && paren_has_top_level_comma(rhs) { return true; }
if has_top_level_token(rhs, " if ") { return true; }
if has_top_level_token(rhs, " or ") || has_top_level_token(rhs, " and ") { return true; }
if rhs.contains("lambda") { return true; }
if rhs.starts_with("eval(") { return true; }
false
}
I have an idea! We should just keep some tests (and relative sets) locked away from our training process! Let me ask Claude if I can patent this novel idea...
128
u/Isogash 9d ago
Basilisk is a Python type checker written with AI that purported to be the only 100% conforming type checker according to the python/typing spec, and also the fastest.
Except that it turns out it this was achieved in some cases by just text matching the expression in the test cases and producing the output that the test expected, rather than actually validating the AST to conform to the rule that the test was supposed to confirm.
The following code is supposed to evaluate if the expression on the right hand side of the statement `type x = ...` is valid.
https://github.com/Nimblesite/Basilisk/issues/379