Infrastructure used to maintain and distribute the Linux operating system kernel was infected for two years, starting in 2009, by sophisticated malware that managed to get a hold of one of the developers’ most closely guarded resources: the /etc/shadow files that stored encrypted password data for more than 550 system users, researchers said Tuesday.

The unknown attackers behind the compromise infected at least four servers inside kernel.org, the Internet domain underpinning the sprawling Linux development and distribution network, the researchers from security firm ESET said. After obtaining the cryptographic hashes for 551 user accounts on the network, the attackers were able to convert half into plaintext passwords, likely through password-cracking techniques and the use of an advanced credential-stealing feature built into the malware. From there, the attackers used the servers to send spam and carry out other nefarious activities. The four servers were likely infected and disinfected at different times, with the last two being remediated at some point in 2011.

Stealing kernel.org’s keys to the kingdom

An infection of kernel.org came to light in 2011, when kernel maintainers revealed that 448 accounts had been compromised after attackers had somehow managed to gain unfettered, or “root,” system access to servers connected to the domain. Maintainers reneged on a promise to provide an autopsy of the hack, a decision that has limited the public’s understanding of the incident.

Besides revealing the number of compromised user accounts, representatives of the Linux Kernel Organization provided no details other than saying that the infection:

• Occurred no later than August 12, 2011, and wasn’t detected for another 17 days
• Installed an off-the-shelf rootkit known as Phalanx on multiple servers and personal devices belonging to a senior Linux developer
• Modified the files that both servers and end user devices inside the network used to connect through OpenSSH, an implementation of the SSH protocol for securing remote connections.

In 2014, ESET researchers said the 2011 attack likely infected kernel.org servers with a second piece of malware they called Ebury. The malware, the firm said, came in the form of a malicious code library that, when installed, created a backdoor in OpenSSH that provided the attackers with a remote root shell on infected hosts with no valid password required. In a little less than 22 months, starting in August 2011, Ebury spread to 25,000 servers. Besides the four belonging to the Linux Kernel Organization, the infection also touched one or more servers inside hosting facilities and an unnamed domain registrar and web hosting provider.

A 47-page report summarizing Ebury’s 15-year history said that the infection hitting the kernel.org network began in 2009, two years earlier than the domain was previously thought to have been compromised. The report said that since 2009, the OpenSSH-dwelling malware has infected more than 400,000 servers, all running Linux except for about 400 FreeBSD servers, a dozen OpenBSD and SunOS servers, and at least one Mac.

Researcher Marc-Etienne M. Léveillé wrote:

In our 2014 paper, we mentioned that there was evidence that kernel.org, hosting the source code of the Linux kernel, had been a victim of Ebury. Data now at our disposal reveals additional details about the incident. Ebury had been installed on at least four servers belonging to the Linux Foundation between 2009 and 2011. It seems these servers acted as mail servers, name servers, mirrors, and source code repositories at the time of the compromise. We cannot tell for sure when Ebury was removed from each of the servers, but since it was discovered in 2011 it is likely that two of the servers were compromised for as long as two years, one for one year and the other for six months.

The perpetrator also had copies of the /etc/shadow files, which overall contained 551 unique username and hashed password pairs. The cleartext passwords for 275 of those users (50%) are in possession of the attackers. We believe that the cleartext passwords were obtained by using the installed Ebury credential stealer, and by brute force.

The researcher said in an email that the Ebury and Phalanx infections appear to be separate compromises by two unrelated threat groups. Representatives of the Linux Kernel Organization didn’t respond to emails asking if they were aware of the ESET report or if its claims were accurate. There is no indication that either infection resulted in tampering with the Linux kernel source code.