前言

前日看到大佬发的机器学习识别XSS的项目代码

• https://github.com/SparkSharly/DL_for_xss

于是搞了一把，记录一下遇到的问题

思路

大佬代码中给出的流程：

• 数据集：GET/POST query
• 范化、分词:将数字串、链接替换为固定词，然后用正则分词
• 特征(word2vec)
• 神经网络(MLP/RNN/CNN)

我实践的流程：

• 范化，分词流程不变
• 特征(分别尝试了word2vec、doc2vec、统计特征)
• SVM

数据预处理

以下是在本地做数据预处理时的 jupyter notebook 代码。

import graphlab as gl

1 取数据

normal = gl.SFrame.read_csv('data/normal_examples.csv')
evil = gl.SFrame.read_csv('data/xssed.csv')

Parsing completed. Parsed 40637 lines in 0.091727 secs.

normal = gl.SFrame(normal)
evil = gl.SFrame(evil)

normal

2 画数据

normal.show()

Canvas is accessible via web browser at the URL: http://localhost:54528/index.html
Opening Canvas in default web browser.

evil.show()

Canvas is updated and available in a tab in the default browser.

evil.head()

3 范化、分词

import nltk
import re
from urllib import unquote

def GeneSeg(payload):
    #数字泛化为"0"
    payload=payload['param'].lower()
    payload=unquote(unquote(payload))
    payload,num=re.subn(r'\d+',"0",payload)
    #替换url为”http://u
    payload,num=re.subn(r'(http|https)://[a-zA-Z0-9\.@&/#!#\?]+', "http://u", payload)
    #分词
    r = '''
        (?x)[\w\.]+?\(
        |\)
        |"\w+?"
        |'\w+?'
        |http://\w
        |</\w+>
        |<\w+>
        |<\w+
        |\w+=
        |>
        |[\w\.]+
    '''
    return nltk.regexp_tokenize(payload, r)

# 增加一列
normal['parsed'] = normal.apply(GeneSeg)
evil['parsed'] = evil.apply(GeneSeg)

normal.head()

evil.head()

4 词表

# 取词的全集生成新的SFrame
ans = []
evil['parsed'].apply(lambda x:[ans.append(i) for i in x])

dtype: list
Rows: 40637
[[None, None, None, None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None], [None, None, None, None, None, None, None], [None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None], [None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None], [None, None, None, None], [None, None, None, None], [None, None, None, None], [None, None, None, None, None], [None, None, None, None, None], [None, None, None, None, None, None, None], [None, None, None, None, None], [None, None], [None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None], [None, None, None, None, None, None, None, None, None], ... ]

evil_word_frame = gl.SFrame(data=ans)

evil_word_frame

# 增加统计列
evil_word_frame['count'] = evil_word_frame.apply(lambda x:len(evil_word_frame.filter_by(x['X1'],'X1')))

# 去重，按数量排序
evil_word_frame = evil_word_frame.unique().sort('count',ascending=False)

evil_word_frame

# 取前300个词作为词表
wordlist_len = 300

evil_word_frame = evil_word_frame[0:wordlist_len]

evil_word_frame

evil_word_frame.save('file/evil_word_list.csv', format='csv')

5 词向量

# 将不在词表内的词全部替换为'WORD'，结果存入'words'列
evil['words'] = evil.apply(lambda x:[i if len(evil_word_frame.filter_by(i,'X1')) else 'WORD' for i in x['parsed']])

evil

normal['words'] = normal.apply(lambda x:[i if len(evil_word_frame.filter_by(i,'X1')) else 'WORD' for i in x['parsed']])

normal

5.1 word2vec

from gensim.models.word2vec import Word2Vec

Using TensorFlow backend.

embedding_size=128
skip_window=5
num_sampled=64
num_iter=100
data_set = evil['words']

data_set[0:10]

dtype: list
Rows: 10
[['search=', '</script>', '<img', 'src=', 'worksinchrome', 'colon', 'prompt', 'x0', '0', 'x0', 'onerror=', 'eval(', 'src', ')', '>'], ['WORD', '<h0>', '<script>', 'alert(', 'hacked', ')', '</script>', '</h0>', 'id=', '0'], ['query=', '<img', '>', '<script>', 'alert(', '"cyb0r_pr0dat0r"', ')', '</script>', '>', 'WORD', 'true', 'ct=', 'null', 'autobounce=', 'true'], ['returnurl=', 'http://u', 'globale', 'searchtext', ')', '>', '<script>', 'alert(', 'xss', 'by', 'm0ke', ')', '</script>'], ['_lang=', '>', '<script>', 'alert(', 'document.cookie', ')', '</script>'], ['language=', '>', '</script>', '>', '>', '<script>', 'alert(', 'arman', 'ires', 'digital', 'boys', 'underground', 'http://u', ')', '</script>'], ['q=', 'bentley', 'stylesheet=', '>', '<script>', 'alert(', "'raven'", ')', '</script>', 'collections=', 'libcms'], ['WORD', 'com_wdshop', 'view=', 'userinfo', 'ajax_json=', 'ajax_fill_city_state', 'format=', 'WORD', 'zip=', '>', '</style>', '</script>', '<script>', 'alert(', 'document.cookie', ')', '</script>'], ['ct_orig_url=', 'arena', '>', '<script>', 'alert(', '0', ')', '</script>', '<iframe', 'src=', 'http://u', '>', '</iframe>'], ['query=', 'search...', 'product=', '>', '</style>', '</script>', '<script>', 'alert(', "'xss'", ')', '</script>', 'page=', '0']]

model=Word2Vec(data_set,size=embedding_size,window=skip_window,negative=num_sampled,iter=num_iter)
#model=Word2Vec(data_set)

5.2 模型存取

model.save('model_word2vec_auto')

# import pickle
# with open('model_word2vec_pickle','wb') as f:
#     pickle.dump(model,f)

model_new = Word2Vec.load('model_word2vec_auto')

# import pickle
# model_new = pickle.load('model_word2vec_pickle')

5.3 数据存取

# evil.save('evil_data')
# normal.save('normal_data')

# evil = graphlab.SFrame('evil_data')
# normal = graphlab.SFrame('normal_data')

5.4 测试模型

embeddings=model_new.wv

embeddings.similar_by_word("</script>",5)

[('keyword=', 0.4176161289215088),
 ('searchfor=', 0.39741984009742737),
 ('page=', 0.38440901041030884),
 ('id=', 0.3829260766506195),
 ('q=', 0.3615732789039612)]

embeddings.similar_by_word("alert(",5)

[(')', 0.3255831003189087),
 ('</marquee>', 0.3212870955467224),
 ('</script>', 0.3093520998954773),
 ('</h0>', 0.30848926305770874),
 ('<h0>', 0.2897389233112335)]

5.5 添加向量和标签

dictionary=dict([(embeddings.index2word[i],i)for i in range(len(embeddings.index2word))])
reverse_dictionary=dict(zip(dictionary.values(),dictionary.keys()))

word2vec={"dictionary":dictionary,"embeddings":embeddings,"reverse_dictionary":reverse_dictionary}

from graphlab import SArray

def generate_vec(words):
    l = SArray([0.0]*128)
    for word in words:
        if word in dictionary:
            l += SArray(embeddings[word])
    return l

# 添加新的vec列、label列
black = gl.SFrame([evil.apply(lambda x:generate_vec(x['words'])),evil.apply(lambda x:1)])
white = gl.SFrame([normal.apply(lambda x:generate_vec(x['words'])),normal.apply(lambda x:0)])

black.show()

Canvas is accessible via web browser at the URL: http://localhost:59854/index.html
Opening Canvas in default web browser.

#black.save('fuck_black')
#white.save('fuck_white')
len(white)

200129

二分类

之后的工作即划分数据集之后训练SVM，这部分工作导出到机器学习平台完成，可能涉敏就不展开了。

反思

• 从结果来看，尽管验证集给出了良好的预测效果，但模型的泛化能力很差。如果想要机器在大量的标签和函数的组合中识别出是否为恶意，则对数据集的要求则会更高。

• 同时，仅根据GET/POST参数判定XSS攻击是不完整的，需关联response的返回内容才能判断是否攻击成功，即便这样也仅能检出反射型XSS。该模型面对真实场景中复杂的输入和触发方式可能会略显鸡肋，需结合业务