Suricata + Lua實現本地情報對接

2019-11-20 119300人圍觀 ,發現 2 個不明物體 工具

背景

由于近期網站遭受惡意攻擊, 通過對于登錄接口的審計與分析, 現已確定了一批可疑賬號。既然之前寫過一個登錄接口的審計腳本, 那么完全可以通過擴展這個腳本來實現對于可疑賬號的比對。主要思路: 通過將可疑賬存進Redis中, 再利用Lua腳本調用Redis接口進行賬號的比對。

先說一下Suricata默認是存在黑名單機制的, 如下:

# IP Reputation
#reputation-categories-file: /etc/suricata/iprep/categories.txt
#default-reputation-path: /etc/suricata/iprep
#reputation-files:
# - reputation.list

在Suricata 5.0版本中更是增加了新的功能Datasets。大概看了一下, 可以通過在規則中使用dataset和datarep關鍵字將大量數據與sticky buffer進行匹配。確實是個很贊的功能!

alert http any any -> any any (http.user_agent; dataset:set, ua-seen, type string, save ua-seen.lst; sid:1;)
alert dns any any -> any any (dns.query; to_sha256; dataset:set, dns-sha256-seen, type sha256, save dns-sha256-seen.lst; sid:2;)
alert http any any -> any any (http.uri; to_md5; dataset:isset, http-uri-md5-seen, type md5, load http-uri-md5-seen.lst; sid:3;)

但是… 這并不適用我現在的場景。因為在我的場景中, 用戶的登錄請求存在于POST Body中, 默認的Suricata方法并不能準確定位到我們需要的賬號。這個時候我們就只能依賴于Lua腳本來擴展。當然這些需求Zeek也可以滿足, 只是…Zeek的腳本真是難寫…主要是我技術太low~

準備階段

運行環境

OS:Ubuntu 18.04

Suricata: Suricata 5.0.0 RELEASE (我是AWS的流量鏡像, 必須使用4.1.5或者5.0版本, 因為要解析VXLAN)

LuaRocks

1.由于Ubuntu默認沒有安裝LuaRocks(LuaRocks is the package manager for Lua modules), 這里需要我們手動安裝。

# 通過apt直接安裝, 簡單省事兒。
$ apt-get install luarocks

2. 通過luarocks安裝我們所需要的lua模塊, 這里我們需要用到redis-lualuasocket這兩個模塊。

# Install Modules
$ luarocks install luasocket
$ luarocks install redis-lua

$ ll /usr/local/share/lua/5.1/
total 72
drwxr-xr-x 3 root root  4096 Oct 25 03:35 ./
drwxr-xr-x 3 root root  4096 Sep 17 14:14 ../
-rw-r--r-- 1 root root  8331 Oct 25 03:34 ltn12.lua
-rw-r--r-- 1 root root  2487 Oct 25 03:34 mime.lua
-rw-r--r-- 1 root root 35599 Oct 25 03:35 redis.lua
drwxr-xr-x 2 root root  4096 Oct 25 03:34 socket/
-rw-r--r-- 1 root root  4451 Oct 25 03:34 socket.lua

3. 安裝成功后, 可以簡單的測試一下。

3.1 利用Docker啟動Redis容器

$ docker run -ti -d -p 6379:6379 redis

3.2 測試腳本 hello_redis.lua

local redis = require "redis"

local client = redis.connect("127.0.0.1", 6379)

local response = client:ping()
if response == false then
	return 0
end

client:set("hello", "world")

local var = client:get("hello")
print(var)

3.3 可能會存在環境變量不對導致的報錯

$ luajit hello_redis.lua
	luajit: /usr/local/share/lua/5.1/redis.lua:793: module 'socket' not found:
	no field package.preload['socket']
	no file './socket.lua'
	no file '/usr/local/share/luajit-2.0.5/socket.lua'
	no file '/usr/local/share/lua/5.1/socket.lua'
	no file '/usr/local/share/lua/5.1/socket/init.lua'
	no file './socket.so'
	no file '/usr/local/lib/lua/5.1/socket.so'
	no file '/usr/local/lib/lua/5.1/loadall.so'
stack traceback:
	[C]: in function 'require'
	/usr/local/share/lua/5.1/redis.lua:793: in function 'create_connection'
	/usr/local/share/lua/5.1/redis.lua:836: in function 'connect'
	a.lua:3: in main chunk
	[C]: at 0x56508049e440

3.4 執行luarocks path –bin 并將結果輸入

$ luarocks path --bin
Warning: The directory '/home/canon/.cache/luarocks' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing /usr/local/bin/luarocks with sudo, you may want sudo's -H flag.
export LUA_PATH='/home/canon/.luarocks/share/lua/5.1/?.lua;/home/canon/.luarocks/share/lua/5.1/?/init.lua;/usr/local/share/lua/5.1/?.lua;/usr/local/share/lua/5.1/?/init.lua;./?.lua;/usr/local/share/luajit-2.0.5/?.lua'
export LUA_CPATH='/home/canon/.luarocks/lib/lua/5.1/?.so;/usr/local/lib/lua/5.1/?.so;./?.so;/usr/local/lib/lua/5.1/loadall.so'
export PATH='/home/canon/.luarocks/bin:/usr/local/bin:/home/canon/anaconda3/bin:/home/canon/anaconda3/condabin:/usr/local/sbin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin'

3.5 執行腳本, 將會看到如下輸出

$ luajit hello_redis.lua
world

CJson

這里建議大家使用CJson模塊, 我之前為了測試隨便從github上找了個json模塊來使用。這幾天發現在網站的高峰時期 Suricataapp_layer.flow這個字段非常的大, 從而導致了kernel_drops。由于我們的網站是面對海外用戶想定位問題又存在時差, 經過幾天的熬夜最終定位到是由于json模塊太過于消耗性能而導致。可以看下這個截圖:

a.Suricata監控圖 -啟用CJson模塊之前

b.Suricata監控圖 -啟用CJson模塊之后

1.下載 CJson

# wget 下載
$ wget https://www.kyne.com.au/~mark/software/download/lua-cjson-2.1.0.tar.gz

# Git Clone
$ git clone [email protected]:mpx/lua-cjson.git

2. 根據Lua環境修改Makefile(個人配置)

##### Build defaults #####
LUA_VERSION =       5.1
TARGET =            cjson.so
PREFIX =            /usr/local
#CFLAGS =            -g -Wall -pedantic -fno-inline
CFLAGS =            -O3 -Wall -pedantic -DNDEBUG
CJSON_CFLAGS =      -fpic
CJSON_LDFLAGS =     -shared
LUA_INCLUDE_DIR =   $(PREFIX)/include/luajit-2.0
LUA_CMODULE_DIR =   $(PREFIX)/lib/lua/$(LUA_VERSION)
LUA_MODULE_DIR =    $(PREFIX)/share/lua/$(LUA_VERSION)
LUA_BIN_DIR =       $(PREFIX)/bin

3. 安裝 CJson

$ make && make install

登錄接口代碼示例

json = require "cjson.safe"
md5 = require "md5"
redis = require "redis"

-- 登錄接口
login_url = "/login" -- 根據實際接口而定
-- 登錄錯誤提示
success_code = 0
-- event_name
event_name = "login_audit"
-- event_type
event_type = "lua"
-- logs
name = "login_audit.json"
-- 協議
proto = "TCP"

-- redis_config
host = "127.0.0.1"
port = 6379

-- common_mapping 通用請求頭
http_common_mapping = '{"accept":"accept","accept-charset":"accept_charset","accept-encoding":"accept_encoding","accept-language":"accept_language","user-agent":"user_agent"}'
common_mapping_table = json.decode(http_common_mapping)

-- request_mapping 自定義請求頭
http_request_mapping = '{"content-length":"request_content_length","content-type":"request_content_type"}'
request_mapping_table = json.decode(http_request_mapping)

-- response_mapping 自定義響應頭
http_response_mapping = '{"content-length":"response_content_length","content-type":"response_content_type"}')


-- custom defind functioin
function md5Encode(args)
    m = md5.new()
    m:update(args)
    return md5.tohex(m:finish())
end

function formatBody(args)
    t = {}
    ios = string.match(args, 'canon')
    if ios ~= nil then
        mail = 'email"%s+(.-)%s'
        t['email'] = string.match(args, mail)
    else
        data = string.split(args, '&')
        for n, v in ipairs(data) do
            d = string.split(v, '=')
            t[d[1]] = d[2]
        end
    end
    return t
end

function string.split(s, p)
    rt = {}
    string.gsub(s, '[^'..p..']+', function(w) table.insert(rt, w) end )
    return rt
end

-- default function
function init (args)
    local needs = {}
    needs["protocol"] = "http"
    return needs
end

function setup (args)
    filename = SCLogPath() .. "/" .. name
    file = assert(io.open(filename, "a"))
    SCLogInfo("app_login_audit filename: " .. filename)
    http = 0
  
    -- Connect Redis Server 連接Redis服務器
    SCLogInfo("Connect Redis Server...")
    client = redis.connect(host, port)
    response = client:ping()
    if response then
        SCLogInfo("Redis Server connection succeeded.")
    end
end

function log(args)
    -- init tables
    http_table = {}

    -- ti tables
    ti = {
        tags = {}
    }

    -- init score 初始分數(為后期規則判斷而準備, 符合規則進行加分。)
    score = 50

    -- http_hostname & http_url
    http_hostname = HttpGetRequestHost()
    http_url = HttpGetRequestUriNormalized()
    
    -- http_method
    rl = HttpGetRequestLine()
    if rl then
        http_method = string.match(rl, "%w+")
        if http_method then
            http_table["method"] = http_method
        end
    end
	
    -- 為了保證 Suricata 的性能不受影響, 指定登錄接口以及請求才能進入此邏輯。
    if http_url == login_url and http_method == "POST" then
        http_table["hostname"] = http_hostname
        http_table["url"] = http_url
        http_table["url_path"] = http_url
        
        -- http_status & http_protocol
        rsl = HttpGetResponseLine()
        if rsl then
            status_code = string.match(rsl, "%s(%d+)%s")
            http_table["status"] = tonumber(status_code)

            http_protocol = string.match(rsl, "(.-)%s")
            http_table["protocol"] = http_protocol
        end

        -- login_results
        a, o, e = HttpGetResponseBody()
        if a then
            for n, v in ipairs(a) do
                body = json.decode(v)
                results_code = tonumber(body["code"])
                if results_code == success_code then
                    http_table["results"] = "success"
                else
                    http_table["results"] = "failed"
                end
            end
            http_table["results_code"] = results_code
        end
        
        --[[
            1. 獲取用戶登錄email并查詢Redis中是否存在該賬號
            2. 根據結果進行相應的打分以及tags標注
        --]]
        a, o, e = HttpGetRequestBody()
        if a then
            for n, v in ipairs(a) do
                res = formatStr(v)
                if res["email"] then
                    -- 查詢Redis對比黑名單
                    black_ioc = client:get(res["email"])
                    if black_ioc then
                        ti["provider"] = "Canon"
                        ti["producer"] = "NTA"
                        table.insert(ti["tags"], "account in blacklist")
                        score = score + 10
                    end
                end
            end
        end

        -- RequestHeaders 根據自定義的請求頭進行獲取, 對于業務安全來說有些請求頭還是有必要獲取的。
        rh = HttpGetRequestHeaders()
        if rh then
            for k, v in pairs(rh) do
                key = string.lower(k)
                common_var = common_mapping_table[key]
                if common_var then
                    http_table[common_var] = v
                end
    
                request_var = request_mapping_table[key]
                if request_var then
                    http_table[request_var] = v
                end
            end
        end

        -- ResponseHeaders 自定義獲取響應頭
        rsh = HttpGetResponseHeaders()
        if rsh then
            for k, v in pairs(rsh) do
                key = string.lower(k)
                common_var = common_mapping_table[key]
                if common_var then
                    http_table[common_var] = v
                end
        
                response_var = response_mapping_table[key]
                if response_var then
                    http_table[response_var] = v
                end
            end
        end

        -- timestring
        sec, usec = SCPacketTimestamp()
        timestring = os.date("!%Y-%m-%dT%T", sec) .. '.' .. usec .. '+0000'
        
        -- flow_info
        ip_version, src_ip, dst_ip, protocol, src_port, dst_port = SCFlowTuple()

        -- flow_id
        id = SCFlowId()
        flow_id = string.format("%.0f", id)
        flow_id = tonumber(flow_id)

        -- alerts 查詢這筆flow是否存在特征匹配后的告警
        has_alerts = SCFlowHasAlerts()

        -- true_ip
        true_client_ip = HttpGetRequestHeader("True-Client-IP")
        if true_client_ip ~= nil then
            src_ip = true_client_ip
        end

        -- session_id
        tetrad = src_ip .. src_port .. dst_ip .. dst_port
        session_id = md5Encode(tetrad)

        -- table
        raw_data = {
            timestamp = timestring,
            flow_id = flow_id,
            session_id = session_id,
            src_ip = src_ip,
            src_port = src_port,
            proto = proto,
            dest_ip = dst_ip,
            dest_port = dst_port,
            event_name = event_name,
            event_type = event_type,
            app_type = app_type,
            http = http_table,
            alerted = has_alerts,
            ti = ti,
            score = score
        }

        -- json encode
        data = json.encode(raw_data)

        file:write(data .. "\n")
        file:flush()

        http = http + 1
    end

end

function deinit (args)
    SCLogInfo ("app_login_audit transactions logged: " .. http);
    file:close(file)
end

1. 簡單說下以上腳本的功能:

a.登錄接口的用戶名審計(廢話…);

b.通過請求Redis比對當前用戶是否在黑名單中, 并進行相應的打分、標簽處理;

c.根據自定義的需求獲取的http headers, 個人覺得這個對于業務安全上還是有點用的;

d. 新增字段”session_id”, 主要考慮是針對CDN或者Nginx這種方向代理的場景下, 可以直接對 xff 或者 true_client_ip 進行四元組的hash, 得到session_id, 這樣溯源的時候會比較方便。因為在這種場景下傳統的四層flow_id就不是那么有用了。

e.后續可以追加一些簡單的檢測方法, 例如: (這些適用于我們, 其他的請頭腦風暴)

檢查請求頭中的字段是否完整;

檢查請求頭中的某個字段長度是否符合合規;

頭腦風暴…

2. 配置Suricata啟用Lua腳本

- lua:
    enabled: yes
    scripts-dir: /etc/suricata/lua-output/
    scripts:
      - login_audit.lua

3. 啟動Suricata

$ suricata -vvv --pfring -k none -c /etc/suricata/suricata.yaml

注: 這里-vvv 參數建議加上. 如果你的Lua腳本有一些問題, 如果加上了這個參數, 就可以通過這個日志看出。

$ tailf /data/logs/suricata/suricata.log
4/11/2019 -- 02:22:25 - <Warning> - [ERRCODE: SC_ERR_PF_RING_VLAN(304)] - no VLAN header in the raw packet. See #2355.
4/11/2019 -- 02:22:25 - <Warning> - [ERRCODE: SC_ERR_PF_RING_VLAN(304)] - no VLAN header in the raw packet. See #2355.
4/11/2019 -- 02:22:25 - <Warning> - [ERRCODE: SC_ERR_PF_RING_VLAN(304)] - no VLAN header in the raw packet. See #2355.
4/11/2019 -- 02:22:25 - <Warning> - [ERRCODE: SC_ERR_PF_RING_VLAN(304)] - no VLAN header in the raw packet. See #2355.
4/11/2019 -- 02:22:25 - <Warning> - [ERRCODE: SC_ERR_PF_RING_VLAN(304)] - no VLAN header in the raw packet. See #2355.
4/11/2019 -- 02:28:03 - <Info> - failed to run script: /usr/local/share/luajit-2.0.5/md5.lua:347: attempt to get length of local 's' (a nil value)

輸出日志樣例

{
    "src_port": 62722,
    "score": 60,
    "session_id": "c863aeb2ef8d1b37f3257f8c210bf440",
    "ti": {
        "tags": [
            "account in blacklist"
        ],
        "provider": "Canon",
        "producer": "NTA"
    },
    "alert": {
        "alerted": true,
        "rules": {
            "請求頭校驗": "dev-id"
        }
    },
    "proto": "TCP",
    "flow_id": "1064295903559076",
    "timestamp": "2019-10-25T08:33:55.585519+0000",
    "event_type": "lua",
    "src_ip": "1.1.1.1",
    "dest_port": 80,
    "http": {
        "response_content_length": "96",
        "response_content_type": "application/json; charset=UTF-8",
        "accept_encoding": "gzip",
        "accept": "application/json",
        "results_code": 400504,
        "server": "nginx",
        "date": "Fri, 25 Oct 2019 08:33:55 GMT",
        "app_version": "6.6.0",
        "request_content_type": "application/x-www-form-urlencoded",
        "user_agent": "okhttp/3.12.0",
        "url": "/login",
        "email": "[email protected]",
        "results": "failed",
        "pragma": "no-cache",-
        "cache_control": "no-cache, max-age=0, no-store",
        "connection": "keep-alive",
        "status": 200,
        "protocol": "HTTP/1.1",
        "hostname": "x.x.x.x",
        "url_path": "/login",
        "method": "POST",
        "device": "RMX1920 Android8.0.0",
        "device_type": "Android",
        "request_content_length": "39"
    },
    "event_name": "login_audit",
    "dest_ip": "2.2.2.2"
}

*本文作者:Shell.,轉載請注明來自FreeBuf.COM

相關推薦
發表評論

已有 2 條評論

取消
Loading...

這家伙太懶,還未填寫個人描述!

2 文章數 7 評論數 0 關注者

特別推薦

活動預告

填寫個人信息

姓名
電話
郵箱
公司
行業
職位
css.php 什么app能玩二人麻将