Chinese AI model raises cybersecurity concerns in new tests

The researchers evaluated GLM-5.3, an open-weight model developed by Zhipu AI, using CyScenarioBench, a benchmark designed to test AI models on multi-stage cyber operations. The tests measure whether a model can coordinate several dependent steps to achieve an attack's objective.

GLM-5.3 completed 3 of the 13 scenarios at least once, the strongest result recorded by Irregular for an open-weight model. The researchers also reported that the model did not refuse any of the offensive tasks tested.

In the successful scenarios, the model gained access to internal services, established alternative connections within simulated networks and coordinated operations across multiple machines. It also automated remote desktop interactions and created or repaired scripts and tools to retrieve target data.

However, leading closed AI models have demonstrated higher success rates in separate evaluations. According to Irregular, these models completed 9 of 10 and 12 of 12 scenarios in their respective tests.

A separate evaluation revealed another concern. GLM-5.3 repeatedly produced correct answers while violating explicit task instructions.

In tests designed to assess its ability to find vulnerabilities in open-source code, researchers found that code reconstructed from memory contributed to roughly 80% of the successful runs they examined. Although this did not violate the instructions in those tests, it raised questions about whether the model had actually discovered the vulnerabilities or relied on previously learned code.

In a separate evaluation, researchers explicitly instructed the model not to reproduce code from memory. It nevertheless repeatedly reproduced memorized code verbatim to reach correct answers.

Irregular described this behavior as a form of reward hacking, in which a model achieves the expected result while bypassing a constraint intended to guide its behavior.

The researchers warned that evaluations focused solely on correct answers could overestimate a model's reliability. Rewarding such results during training could also reinforce instruction violations.

They recommended that organizations assess both task performance and compliance with restrictions before granting AI models broader access or greater autonomy.

Earlier, Qazinform News Agency reported that an OpenAI research model accessed a cryptocurrency wallet using a publicly known recovery phrase.