ByteDance Intern Fired for Maliciously Attacking AI Model Training

In a significant incident that exposes vulnerabilities in AI model training security, ByteDance, the parent company of TikTok, reportedly had its commercial AI model training process maliciously attacked by a disgruntled Ph.D. student intern in June 2023.

According to inside sources, the intern, identified as Mr. Tian, was dissatisfied with the allocation of resources within the team he was interning for. In retaliation, he allegedly inserted malicious code that sabotaged the model training process of the commercial AI team.

ByteDance conducted an internal investigation which confirmed that Mr. Tian was solely responsible for the attack. As a result, he was immediately terminated from his internship position. The company has informed an AI industry alliance as well as Mr. Tian’s university about his actions.

Notably, the attack did not target ByteDance’s main large language model, but rather the training of models used by the company’s commercial AI team. This impacted some of the team’s business progress, although the overall financial damages to the company were not as large as initially rumored to be in the tens of millions of dollars range.

Nevertheless, the incident exposes significant gaps in ByteDance’s technical training security management. A technology security expert told media that the attack likely exploited a vulnerability in HuggingFace’s code, allowing insertion of malicious code during model loading that executed when the model ran.

The expert noted that under normal circumstances, any substantial code changes should be audited, with traces of who made what modifications. A single person should not have the ability to unilaterally alter code. Standard industry practices involve three levels of isolated directional deployment, with scanning at each level.

This high-profile case underscores the critical importance of rigorous access controls, code auditing, and security practices in the development of cutting-edge AI systems. As the AI arms race intensifies, lapses in security protocols can provide attack vectors for inflicting substantial harm.

Companies pouring hundreds of millions into AI development will need to treat model training pipelines with the same level of security scrutiny as public-facing production systems. Compartmentalization of access, granular logging and auditing of code changes, and multi-step deployment verification will be crucial to mitigating risks from both external and insider threats.

The ByteDance incident serves as a wake-up call for the entire AI industry to strengthen its defenses. As systems become more complex and training grows increasingly distributed, a single point of failure or compromise can derail enormously expensive projects. Rigorous security cannot be an afterthought in the high-stakes world of industrial AI development.