Abstract: The linear transformation of the block cipher Serpent is studied in detail through the introduction of a new concept called "influence set of a bit", and another concept called "Λ Set based on a nibble" is also introduced. In virtue of these two concepts, a modified Square attack based on a nibble is designed to attack on the reduced Serpent of 3 rounds with 16 chosen plaintext ciphertext blocks and average 2127 inverse S box operations. The detailed attack steps are as following: ① Finding ciphertext blocks corresponding to the 16 plaintext blocks satisfying some conditions given in the paper; ② Arbitrarily taking 128 bit key K 3 , and evaluating the exclusive OR’s of this key and the 16 ciphertext blocks given above; ③ Finding the inverse S 2 operations on the resulting 16 blocks; and ④ Finally evaluating the bit wise exclusive OR’s of the 16 blocks obtained in ③ above; if the result is 0, then K 3 is the right key, otherwise return to step ②.