For details on the way A3 and A8 are used see Authentication Center.
A3 and A8 both take a 128-bit key (Ki) and a 128-bit challenge (RAND) as inputs. A3 produces a 32-bit response (SRES) and A8 produces a 64-bit session key (Kc). A3/A8 is the combined function with Ki and RAND as inputs and SRES and Kc as outputs.
As A3 and A8 are not further specified, operators can freely choose the concrete algorithms used for A3 and A8.
The COMP128 algorithms implement the A3/A8 function. There are three of them:
All of them are built around a compression function with two 128 bits inputs and one 128 bits output, hence their names. Ki and RAND are used as the inputs of the compression function. Bits from its output are then used to fill SRES and Kc.
COMP128-1 uses a compression function with eight rounds which is based on a butterfly structure with five stages. SRES is filled with the first 32 bits of the output. Kc is filled with the last 54 bits of the output followed by ten zeroes.
For a full description of the algorithm, the reader can view the OsmocomBB implementation.
The implementation of COMP128-2 and COMP128-3 is noticeably more complex than COMP128-1. For a full description of the algorithm, the reader can view the OsmocomBB implementation or FreeRADIUS implementation, both based on the Python code from the Secrets of Sim3 article. COMP128-2 is identical to COMP128-3 except for the fact that at the end, it clears the 10 rightmost bits of Kc.
The COMP128-1 hash function is considered weak because there is insufficient diffusion of small changes in the input. Practical attacks have been demonstrated that can recover the subscriber key from the SIM.4
The session keys produced by COMP128-1 and COMP128-2 intentionally have only 54 bits of entropy. This significantly weakens the A5 or A6 encryption.
Briceno, Marc; Goldberg, Ian; Wagner, David (1998), Implementation of COMP128, archived from the original on 2009-03-18 https://web.archive.org/web/20090318143444/http://www.scard.org/gsm/a3a8.txt ↩
Tamas, Jos (2013), Secrets of the SIM, archived from the original on 2014-12-24, retrieved 2014-12-24 https://web.archive.org/web/20141224034734/http://www.hackingprojects.net/2013/04/secrets-of-sim.html# ↩
Brumley, Billy (2004), A3/A8 & COMP128 (PDF) http://www.tcs.hut.fi/Studies/T-79.514/slides/S5.Brumley-comp128.pdf ↩