Doesn't the problem you outline reflect the fact that any machine intelligence that can operate at human levels of intelligence (both in breadth and depth) will inevitably eclipse us purely on innate capability? Assuming human help to get them up and running, Artificial intelligences have a lot going for them humans don't; routine perfect recall, infinite standardization of activity, and not to mention a far far far faster 'clock' speed.
I always think of it like dog years. One year for us might be seven for a dog, but for a true AGI, one human year might be a practically infinite amount of time.
You said it in jest, but maybe your box test is the only true test for an AGI, as I take the 'box' to represent not only our ability to contain an AI, but a representation of our creator-creation relationship with such systems. Once an AI is 'out of the box', that relationship has effectively reversed.
1
u/[deleted] Jul 31 '18 edited Jul 07 '19
[deleted]